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1. Introduction 

An important problem in the theory of automorphic forms is to understand peri¬ 
ods, and how they are related with L-functions and their special values, as well as 
with functorial liftings. A prototypical example for this is the connection between 
symmetric and exterior square L-functions, functorial liftings from classical groups 
to GLn, and certain periods, which dates back at least to |JS90j . and is more fully 
explicated in |GRS97] . and |GRS11] . Some more exotic examples are found, for ex¬ 
ample, in |GH04j . [GH07] . and a general framework which extends beyond classical 
groups is discussed in |GJS10] . 

The connection between poles and liftings is well- understood, at least philosoph¬ 
ically: one expects that the L function attached to a generic cuspidal representation 
TT and a hnite dimensional representation r of the relevant L-group will have a pole 
at 1 if and only if the stabilizer of a point in general position for the representation r 
is reductive and tt is in the image of the functorial lifting attached to the inclusion of 
the stabilizer of such a point. For example, in the exterior square representation of 
GL 2 n(C), the stabilizer of a point in general position is Sp 2 n{C), so one expects that 
a pole of the exterior square L function indicates cuspidal representations which are 
lifts from S02n+i- The connection with periods is up to now less well understood. 

In order to prove the expected relationship between poles and liftings in specihc 
examples, and in order to draw periods into the picture, it is useful, perhaps essential, 
to have some sort of an analytic handle on both the L-function and the lifting. An 
analytic handle on the L-function may be provided by an integral representation, 
either of Langlands-Shahidi type or otherwise (integral representations which are 
not of Langlands-Shahidi type are often termed “Rankin-Selberg”). An analytic 
handle on the lifting may be provided by an explicit construction. 

Integral representation of L functions and explicit construction of liftings between 
automorphic forms on different groups are important subjects in their own right as 
well. For example, integral representations are, as far as we know, the only way to 
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establish analytic properties of L functions in new cases. When an integral represen¬ 
tation produces L functions whose analytic properties are already well understood, 
it nevertheless provides a new insight into the connection with periods, and identi¬ 
ties among periods which can be otherwise quite surprising. This is the case in the 
present paper. 

For explicit construction of liftings, there are two main ideas we know of. Each is 
related to the other and both are related to the theory of Fourier coefhcients attached 
to nilpotent orbits f |GRS03j . |G06j ). 

The hrst main idea is to use a “small” representation as a kernel function. The 
prototypical example of this type is the classical theta correspondence f |H79j ). In 
this type of construction, an automorphic form, which is dehned on a large reductive 
group H is restricted to a pair of commuting reductive subgroups, and integrated 
against automorphic forms on one member of the pair to produce automorphic forms 
on the other. In general, there is no reason such a construction should preserve 
irreducibility, much less be functorial. The right approach seems to be to take auto¬ 
morphic forms on H which only support Fourier coefhcients attached to very small 
nilpotent orbits. For example, a theta function, dehned on the group 5'p4^„(A) only 
supports Fourier coefhcients attached to the minimal nilpotent orbit of this group. 
Its restriction to 5'p2n(A) x 02m (A) provides a kernel for the theta lifting between 
these groups. Functoriality of this lifting was established in |R82j . This method has 
enjoyed brilliant success, but also has signihcant limitations. For example, it is not 
at all clear how the classical theta correspondence could be extended to other groups 
of type Cn X Dm'- the embedding into Sp^mn is specihc to Sp 2 n x 02m- The results of 
this paper hint at a possible way around this difficulty. 

The second main idea in explicit construction of correspondences is the descent 
method of Ginzburg, Rallis, and Soudry f |GRSllj . see also [HSj b This construction 
treats the Fourier coefficients themselves essentially as global twisted Jacquet mod¬ 
ules, mapping representations of larger reductive groups to representations of smaller 
reductive groups. As before, in general there is no reason for this construction to re¬ 
spect irreducibility, much less be functorial, and a delicate calculus involving Fourier 
coefficients seems to govern when it is. 

In this paper we dehne and study two new multi-variable Rankin-Selberg integrals, 
which are dehned on the similitude orthogonal groups GSO 12 and GSOig- These 
integrals are similar to those considered in |BG98j . |G96j . |G95j . |GH15j . |GH13j . in 
that each involves applying a Fourier-Jacobi coefficient to a degenerate Eisenstein 
series and then pairing the result with a cusp form dehned on a suitable reductive 
subgroup. To be precise, GSO^n has a standard parabolic subgroup Q whose Levi 
is isomorphic to GL 2 n x GS02n- The unipotent radical is a two step nilpotent group 
and the set of characters of the center may be thought of as the exterior square 
representation of GL 2 n twisted by the similitude factor of GS02n- The stabilizer of 
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a character in general position is isomorphic to 

C* := {{91,92) e GSp2n X GSO^n : K9i) = K92^)}- 

Here, A denotes the similitude factor. The choice of a character in general position as 
above also determines a projection of the unipotent radical onto a Heisenberg group 
in 4n^ + 1 variables, and a compatible embedding of C into S'p 4 „ 2 . 

Our Fourier-Jacobi coefficient dehnes a map from automorphic functions on GSO^ni^) 
to automorphic functions on C'(A). In the case n = 2 and 3 we apply this coefficient 
to a degenerate Eisenstein series on GSOQn{^) induced from a character of the par¬ 
abolic subgroup P whose Levi factor is isomorphic to GL^ x GL^n-s x GLi. We then 
pair the result with a pair of cusp forms dehned on GS'p 2 n(A) and GS 02 n{^) respec¬ 
tively. The results suggest an intriguing connection with the theta correspondence 
for similitude groups. 

Indeed, in the case n = 2, the global integral turns out to be Eulerian, and to give 
an integral representation of 

L'^(si,n X ri)L‘^(s2,n X Ts), 

where H is a generic cuspidal automorphic representation of GSp 4 {A) and ri,r 2 
are two (generic) cuspidal automorphic representations of GL 2 {A) having the same 
central character, so that ri®r 2 is a (generic) cuspidal automorphic representation of 
GS 04 ^{A). It follows that the original integral has poles along both the plane Si = 1 
and the plane S 2 = 1 if and only if H is the weak lift of Ti ® T 2 corresponding to the 
embedding 

GSpin 4 ^{C) = {{ 91 , 92 ) £ GL{2,C)^ \ det 91 = det 92 } ^ GSpin^{€,) = GS'p 4 (C). 

It is known that the functorial lift corresponding to the embedding 

SOi{C) ^ SO^{€) 

is realized via the theta correspondence. Our Eulerian integral suggests that the 
Fourier-Jacobi coefficient of the double residue of our Eisenstein series provides a 
kernel for the theta correspondence for similitude groups. This is particularly intrigu¬ 
ing since the Fourier-Jacobi coefficient construction extends directly to any group of 
type D^ri, whereas there seems to be no hope of extending the theta correspondence 
to any representations of such groups which do not factor through the orthogonal 
quotient in any direct way. 

The integral corresponding to n = 3 provides some more evidence for a connection 
with the theta correspondence, in that the global integral unfolds to a period of 
GSpQ which is known to be nonvanishing precisely on the image of the theta lift 
from GSOe ([GRS97]). 

We now describe the contents of this paper. In section [2] we hx notation and 
describe a family of global integrals, indexed by positive integers n. In section E] we 
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unfold the global integral corresponding to the case n = 2, obtaining a global integral 
involving the Whittaker function of the cusp form involve which, formally, factors as 
a prodnct of local zeta integrals. These local zeta integrals are stndied in sections El 
ElCl after certain algebraic results required for the unramihed case are established 
in section 01 Once the local zeta integrals have been studied we return to the global 
setting for sections [S] and 01 where we record the global identity relating the original 
zeta integral and L'^(si,n x ri)L'^(s 2 ,n x T 2 ),, and deduce a new identity relating 
poles of these L functions and periods. Finally, in section [TUI we briefly describe 
what happens in the case n = 3, omitting details. We remark that the case n = 1 
is somewhat degenerate, as the split form of GSO 2 is a torus; our global integral 
appears to vanish identically in this case. 

This work was nndertaken while the anthors were visiting ICERM for a program on 
Antomorphic Forms, Combinatorial Representation Theory and Multiple Dirichlet 
Series. They thank ICERM and the organizers. 

2. Notation 

Write Jn for the matrix 

1 \ 

a y 

If is an nXm matrix, write *-g for the transpose of g and tg for the “other transpose,” 
Jm^gJn- Let g* = tg~^- Let G = GSOn denote the identity component of GOn '■= 
{g e GLn : gJn^g e GLi ■ J„}. If n is odd, then GOn is the prodnct of SOn and 
the center of GLn- If n is even, then GSOn is the semidirect product of SOn and 
{diag(A/|,Is) : A e GLi). Here Ik is the kxk identity matrix. The group GSOn has 
a rational character A : GSOn GLi, called the similitude factor, such that 

gJjg = \{g)-Jn, (g^GSOn). 

The set of upper triangular (resp. diagonal) elements of GSOn is a Borel subgroup 
(resp. split maximal torus) which we denote BcsOn (resp. TcsOn)- parabolic 
(resp. Levi) subgroup will be said to be standard if it contains BcsOn (resp. TcsOn)- 
The unipotent radical of Bgso„ will be denoted U. We number the simple (relative 
to BGS 02 n) roots of TGS 02 n in G ai,..., On so that = talU+Gi+i for 1 < i < n - 1 , 
and = tn-i,n-iltn+i,n+i- Here, we have used the exponential notation for rational 
characters, i.e., written instead of a{t) for the value of the root a on the torus 
element t. 

Dehne mp : GL 3 x GL^n-i) ^ GLi into GSO%n by 

nrp(5'i,5'2,A) ^ diag{\gi,Xg2,gl,gl). 


( 1 ) 
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Denote the image by Mp. It is a standard Levi subgroup. Let P be the corresponding 
standard parabolic subgroup. Thus, P = Mp xUp, where Dp is the unipotent radical. 
We use ([fj) to identify Mp with GL^ x GL^n-z ^ GLi. 

Recall that a character of F^\K^ (i.e., a character of PP trivial on F^) is normalized 
if it is trivial on the positive real numbers (embedded into PP diagonally at the hnite 
places). An arbitrary quasicharacter of may be expressed uniquely as the 

product of a normalized character and a complex power of the absolute value. If 
X = (Xd X 25 Xs) is a triple of normalized characters of F'^\PP and s = (si, S 2 , S 3 ) e C^, 
write (x; s) for the quasicharacter Mp(A) ^ C by 

(x;s)(fi'i,5'2, A) := Xi(det5fi)|det5fi|^W2(det 5(2)1 det5(2|*W3(A)|A|^L ( 2 ) 

Then (x;s)(aJ6n) = (x; s)(mp(a-i/ 3 ,a-i/ 3 („_i), a^)) = xl^xi"^''xi(a)|a| 2 * 3 - 3 ^i+( 3 - 3 n). 2 . 
The pullback of (x;s) to a quasicharacter of T’(A) will also be denoted (xis). 

Consider the family of induced representations /n(ip^^^(x; s) (non-normalized in¬ 
duction), for hxed x and s varying. Here, we £x a maximal compact subgroup K of 
G{A) and consider iX-£nite vectors. The map /n(ip|^^(x; s) 1 -^ s gives this family the 
structure of a fiber bundle over C^. By a section we mean a function x G(A) ^ C, 
written {s,g) 1 -^ fx-,s{g), such that /^;s e /n(ip^^^(x;s) for each s e C^. A section 
is flat if the restriction of f^-g to K is independent of s. Write Flat(x) for the space 
of flat sections. 

For such a function f^.g, let E(fy..g,g) be the corresponding Eisenstein series, de- 
hned by 

E{fx-,s,9)= E fxAig) 

y^P(F)\G(F) 

when this sum is convergent and by meromorphic continuation elsewhere. The sum is 
convergent for Re(si -S 2 ) and Re(s 2 ) both sufficiently large. (Cf. |MW95] . §11.1.5.) 

Let Q = Mq X Uq be the unique standard parabolic subgroup of G, such that 
Mq = GL 2 n X GS02n- We identify Mq with GL 2 n x GS02n via the isomorphism 

'mQ{gi,g2)-=diag{\{g2)gi,g2-,gl), (^1 e GL 2 n, g2^GS02n)- 

The unipotent radical, Uq, oi Q can be described as 


^ l2n 

a: 

Y 

Z' \ 


0 

L 

0 

-iY 


0 

n 

0 

In 

t 

-w 

:Z'^-XtY^-YtX+ t^' = 0 - 

0 

0 

0 

^2n j 

> 


( 3 ) 
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Let ^A 2 n := {Z e Mat 2 nx 2 n • tZ = -Z}. Then we can define a bijection (which is not a 
homomorphism) uq : Mat 2 nxn x Mat 2 nxn Uq by 



^271 

X 

Y Z- 

-I(XtY + Y,X)\ 


uq{X,Y,Z) = 

0 

0 

In 

0 

0 

In 

1 1 

, X,jY G M8.t2nx'n? ^ ^ ^2n 


1 0 

0 

0 

hn ) 


Then 






uq{X,Y, 

Z)uq{U, 

II 

uq{X + U,Y + V, 

Z^W-{X,V)^{U,Y)), 


(X, Y,U,V e Mat„x 2 n, Z,W e 2a 2„), where 

{A, B) := AtB - B fA, (x4, i? e Matnx2n)- 

It follows that uq{X,Y, Z)~^ = uq{-X,-Y,-Z), and that if [x,y] = xyx~^y~^ de¬ 
notes the commutator, then [mq(X, 0,0), ^^(OjyjO)] = uq{0,0,-X tY + YtX) = 
uq{0, 0, {Y,X)). Define 1{Z) = Tr{Z ■ diag{In, 0)) = rr e Z define 'H 2 n+i 

to be G” X G" X Ga equipped with the product 

{xi,yi,zi){x2,y2,Z2) = ixi+X2,yi + y2,zi +Z2 + xity2-yitX2)- 

Write r for the map from Mat 2 nxn to row vectors corresponding to unwinding the 
rows: r(X) := ... Xi^n,X2,i, ■ ■ ■ X2n,n, and write r' for the similar map which un¬ 

winds the rows and negates the last n. Explicitly: 

^ (^) — X _Y^ ) — (yi,l, ■ ■ ■ yi,ni 2/1,1) • • • yn,ni ~2/n+l,l) • • • ) ~2/2n,n) 

for y = (^ Mat 2 nxn) Yi,Y 2 e Mat„xn • Then we can define a homomorphism from 
Uq to 'H 4 „ 2 +i, the Heisenberg group with 4n^ + 1 variables, by 

j{uq{X,Y,Z)) = {r{X)y{Y),l{Z)). 

The stabilizer of I in Mq is 

Gq := {GSp2n X G^02.)° = {{gu92) e GSp2n X GS02n \ X{gi) = A(^2)-'}, 

where X{gi) is the similitude of gi. For any subgroup H of GSp 2 n x GS02n, let 
H° := H n Cq. The kernel of / is a Gg-stable subgroup of Uq, and is also equal 
to the kernel of j. Note that X{g 2 ) is also the similitude factor of {gi,g 2 ) as an 
element of GSO^n, and that the center of Gq is equal to that of GSOqu- Define 
T = TcsOn Gq, B = BcsOn ri Cq and = 27 n Cq. They are a split maximal torus, 
Borel subgroup, and maximal unipotent subgroup of Cq, respectively. 

The group of automorphisms of whose restrictions to the center of T-LirP+i 

are the identity is isomorphic to S'p 4 „ 2 . Identifying the two groups defines a semidi- 
rect product 5'p4„2 x Let Rq = Cq x Uq. The homomorphism j - Uq ^ 
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extends to a homomorphism Rq Sp^n'^ x R^n'^+i- Indeed, for each c e Cq, the auto¬ 
morphism of Uq defined by conjugation by c preserves the kernel of j, and therefore 
induces an automorphism of R/^rP+i- Moreover, this automorphism is identity on 
the center of '^4^2+1 because c fixes 1 . This induces a homomorphism Cq ->■ S'p4„2, 
which we denote by the same symbol j, and which has the dehning property that 
j{cuc~^) = j{c)j(u)j(c)~^ for all c e Cq and u e Uq. We may then regard the two 
homomorphisms together as a single homomorphism (still denoted j) from Rq to 

For a positive integer M, identify the Siegel Levi of Sp 2 M with GLm via the 
map 9 - It acts on R2M+1 by g{x,y, z)g~^ = (xg~^,yg, z). Note that for 

gi € GSp2n and 92 e GS02n, the matrix niQ(91,92) e Mq maps into GL2rfl if and 
only if it normalizes {uq(W, 0 , 0 ) : X e Mat2nxn}, he., if and only if 92 is of the form 

( 93 ) ^ 

mQ(9i,92) ^,j), (gi e GSp2n, 92 ^ GLn). (4) 

Then 

^Qigi,92)uQ(X,0,0)m}^(g^^, 92^) = UQ(giXg2^,0,0), (Vg^ e GSp2n, 92 e GLn), 
and j(mQ(gi, 92)) e GT2„2 c Sp^n"^ is the matrix satisfying 

r(X)j(m}Q(gi,g2)) =r(gl^Xg2). 

The determinant map G'T2„2 ^ GLi pulls back to a rational character of this sub¬ 
group of Cq which we denote by det. Thus det(mQ(gi,g2)) = det det (7I” = 
det for gi e GSp2n,g2 ^ GLn- On T c Cq, the rational character det 
coincides with the restriction of the sum of the roots of Tcsoer. in {uQ( 0 ,y, 0 ) : Y € 
M!at2nxn}' • 

Let be a additive character on F\A and 'ipi(Z) := ?/) o /. The group Riri'^+i(A) 
has a unique (up to isomorphism) unitary representation, a;^, with central character 
'll), which extends to a projective representation of '^ 4 ^ 2+4 (A) x Sp 4 n 2 (A) or a genuine 
representation of 'H4„2+i(A) x S'p4„2(A), where Sp^^^iA) denotes the metaplectic 
double cover. 

Lemma 2.1. The homomorphism j : C'q(A) ^ Sp 4 n^(A) lifts to a homomorphism 
Cq(A) ^ Fp4„2(A). 

Proof. Write pr for the canonical projection Sp 4 ^ 2 (A) S'p4„2(A). We must show 

that the exact sequence 

1 ^ {±1} ^pr-fyi(C'Q(A))) ^^(^^^(A)) -> 1 
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splits, i.e., that the cocycle determined by any choice of section is a coboundary. The 
analogous result for Sp2n x S02n, over a local held is proved in 0. corollary 3 . 3 , 
p. 36 , or [P 93 ] . lemma 4 . 4 , p. 12 . The extension to Cq follows from section 5.1 of 
|HK 92 j . The global statement then follows from the corresponding local ones. □ 

Thus we obtain a homomorphism Rq{A) S'p4„2(A) ix '^4,12+1 (A) which we still 
denote j. Pulling back through j produces a representation of Rq{A) which we 
denote This representation can be realized on the space of Schwartz functions 
on Mat2nxn(A) with action by 

[oj^,i{uQ( 0 , 0 ,Z)).(j)] ='0z(Z)0 [cu^,z(mq(A:,O,O)).0](O =(j)(^ + X) 

(m^ (^1, ^2 ) ) -0] (0 = 7 v.,det (gi ,92) I ^et (^1 , C/2 ) | ) • 

(Cf. |GR,Sllj . p. 8.) Here denotes the Weil index. The representation has 
an automorphic realization via theta functions 

e{(j),ug):= ^ [uj^(ug).(j)]{ 0 , (we ^4n2+i(A), ,Sp4„2(A)). 

^eMat2nxn(E) 

Here 0 e 5(Mat2nxn(A)) (the Schwartz space of Mat2nxn(A)), 'H4„2+i(A) is identihed 
with the quotient of Uq by the kernel of /, and is identihed with the subgroup 

of its automorphism group consisting of all elements which act trivially on the center. 

Lemma 2.2. Consider the Weil representation o/'H 2 n+i(A) x Sp2niA) o^nd its auto¬ 
morphic realization by theta functions. Let V be a subgroup ofR2n+i which intersects 
the center Z trivially. Thus V corresponds to an isotropic subspace of the symplectic 
space R2n+ilZ- Let V-‘- = {v' e R2n+ilZ ■ {v',V) = 0 } =i H, and let Py be the parabolic 
subgroup of Sp2n which preserves the flag 0 c P c f/-*- c R2n+ilZ. Note that the Levi 
guotient of Py is canonically isomorphic to GL(V) x SpiV^jV). Thus Py has a pro¬ 
jection onto the group GLi x SpiV^jV) induced by the canonical map onto the Levi 
guotient and the determinant map det: GL(V) GLi. The function 



is invariant by the A-points of the kernel of this map on the left. (Throughout this pa¬ 
per, if H is an algebraic group defined over a global field F, then [H] := H(F)\H{A).) 


Proof. First assume that V is the span of the last k standard basis vectors for some 
k <n. Then 



, 0 ,v, 0 )g).f)]{C)dv= Y. [u;,/,(^). 0 ](O,... , 0 ,f). 
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and invariance follows easily from the explicit formulae for given, for example on 
p. 8 of |GRSllj . The general case follows from this special case, since any isotropic 
subspace can be mapped to the span of the last k standard basis vectors, for the 
appropriate value of /c, by using an element of Sp^n^F). □ 

For e Indp|^j(x;s), and (p e 5(Mat2nxn(A)), let 

E^^'^\fx-,s,9)= f du E{f^.s,ug)e{(p,j{ug)), (g^CgiA)). (6) 

-'[Uq] 

Recall that Cq was identified above with a subgroup of GSp2n x GSO^n- If I? ^ Gq 
then gi will denote its GSp2n component and g2 will denote its GS02n component. 
Now take two characters uji,ijJ2 '■ C, and two cuspforms (pi, dehned on 

GSp2n{A) and ipi, defined on GS02niA), such that ipi{a- g) = Ui{a)ipi(g), for z = 1 
or 2, a € A’', and g e GSp2niA) or GS02n{,A) as appropriate. Choose Xi,X2,X3 so 
that x7^X2^X3Wi^W 2 is trivial, and consider 

= f E^^'^\f^,s,g)pi{gi)p{g2)dg. (7) 

J Z(A)Cq(F)\Cq(A) 

To simplify the notation, we may also treat the product (pi(p2 as a single cuspform 
defined on the group Cq, and write (p{g) = (pi(gi)(p2ig2), and Note 

that the integral converges absolutely and uniformly as s varies in a compact set, 
simply because is of moderate growth, while ipi and ip2 are of rapid decay. 


3. Global Integral for GSO 


12 




X2^ 


^98 

97^ 



2:2 

Z 3 

0 

A = 

X 3 

X4 

, = 

ye 

95 

Z = 

Z 4 

2:5 

0 

-^3 


X 3 

Xq 


2/4 

93 



0 

~Z5 

-^2 



xs ) 


[92 

91 1 



- Z 6 

-Z 4 

-^1/ 


In this section we consider a global integral ([ 7 ]) in the case n = 2 . Thus G = GiS'Oi2. 
If uq{X,Y, Z) is an element of Uq, we fix individual coordinates as follows: 


( 8 ) 


Theorem 3.1. Forn in the maximal unipotent subgroup N let ipisi{n) = "0(^12+ ^^23~ 
+ Rsr), and let 

= / p{ng)'ipN{n)dn. (9) 

J[Af] 

Let U4 be the codimension one subgroup of N defined by the condition n23 = n^Q. For 
0 6 i5(Mat4x2(A)), € i?Q(A), write 

(a b\ 

1 0 


W^9)= / da db[u^{g)(j)] 
JA^ 


0 1 

VO 0/ 


'tp{-a). 


( 10 ) 
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Finally, let w be the permutation matrix attached to the permutation 

/l 2 3 4 5 6 7 8 9 10 11 12 \ . . 

^7 10 11 12 4 5 8 9 1 2 3 oj’ 

and let Uq = t/g n w~^Pw. Then the global integral ([ 7 ]) is egual to 

/ / fxA'^^9)h{4>,ug) dudg. (12) 

J Z(A)U4{A)\Cq{A) ^ Ju^{A)\Uq{A) 

Remark 3.2. The permutation matrix w represents an element of the Weyl group 
of G relative to Tq- We also record an expression for w as reduced product of simple 
reflections. We also introduce some notation for elements of the Weyl group. We 
write w\i\ for the simple reflection attached to the simple root Ui, and w[iii2 • • - 4] 
for the product w[i{\w[i2\. ■ ■ w[ik\. Then w = t(;[ 64321465432465434654 ]. 

Before proceeding to the proof, we need to know the structure of the set P\GIRq. 

3 . 1 . Description of the double coset space P\GIRq. Clearly, the identity map 
G ^ G induces a map pr : P\GIRq P\GIQ. Each element of P\GIQ contains 
a unique element of the Weyl group which is of minimal length. Recall that the 
group of permutation matrices which are contained in G maps isomorphically onto 
the Weyl group of G. A Weyl element of minimal length in its P, Q double coset 
corresponds to a permutation cr : {1,..., 12} {1,..., 12} such that 

• cr(13 - z) = 13 - cr(z), Vz, 

• cr is an even permutation. 

• If 1 < z < j < 4 , 5 < z < j < 8, or 9 < z < j < 12 , and if {i,j} {6, 7 }, then 

• If 1 < z < j < 3 , 4 < z < j < 6, 7 < z < j < 9 , or 10 < z < j < 12 , then 
cr-i(z) < 

Such a permutation a is determined by the quadruple 

(#({ 1 , 2 , 3 , 4 } n zt-H{ 3 z - 2 , 3 z - l, 3 z})))ti. 

Deleting any zeros in this tuple gives the ordered partition of 4 corresponding to the 
standard parabolic subgroup •= GL^ n a~^Pcr. (Here we identify the permutation 
a with the corresponding permutation matrix, which is in GSOu, and identify g € 
GL4 with dia,g(g, R, g*) e GSOu.) Now, for any parabolic subgroup Po of GSO4, 
we have GSO^ = P0SO4. It follows that g crdiag(g',/4,5f*) induces a bijection 
P(j\GLilGSpi pr“^(F ■ a ■ Q) c P\GIRq. Therefore we must study the space 
P'\GL4lGSp4,, where P' is an arbitrary parabolic subgroup of GL4. 

Lemma 3.3. Let S be a subset of the set of simple roots in the root system of type 
A3. Let Ps,Ps denote the standard parabolic subgroups GL4, and SOq, respectively, 
corresponding to S. Then PsXGL^jGSp^ and PASOq/SO^ are in canonical bijection. 
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Proof. This follows from considering the coverings of SOq and GL^ by the group 
GSpiriQ which are described in |HS] and section 2.3 of |AR11] . respectively. The 
preimage of SO^ in GSpin^ is GSpin^ = GSp^. Since the kernels of both projections 
are contained in the central torus of GSpirit, which is contained in any parabolic 
subgroup of GSpiriQ it follows that both Ps\GLjGSpi and Pf\S 0 Q/S 05 are in 
canonical bijection with Pg\GSpinQlGSpin^, where P'^ is the parabolic subgroup of 
GSpiriQ determined by S. □ 

Now, in considering SO^jSO^, we embed SO^ into SOq as the stabilizer of a fixed 
anisotropic element vq of the standard representation of SOq. Then Pg\SOQ/SOQ 
may be identified with the set of P^-orbits in SOq ■ vq. For concreteness, take SOq 
to be defined using the quadratic form associated to the matrix Jq, and take vq = 
*[0,0,1,1,0,0]. The SOq orbit of Vq is the set of vectors satistfying ^v- Jq-v = hjQ- Jq- 
V = 2. Note that each of the permutation matrices representing a simple reflection 
attached to an outer node in the Dynkin diagram maps Vq to Vi := *[0,1,0,0,1,0], 
and that a permutation matrix representing the simple reflection attached to the 
middle node of the Dynkin diagram maps vi to V 2 ■= *[1,0,0,0,0,1]. 

Lemma 3.4. Number the roots of SOq so that a 2 is the middle root. (This is not 
the standard numbering for SOq, but it matches the standard numbering for GL^, 
and the numbering inherited as a subgroup of GSOu-) Write V for the standard 
representation of SOq. The decomposition of SOq-Vq into Pf orbits is as follows: 


s 

orbit reps in V 

double coset reps 

0 

V0,Vl,V2 

1 

of 

1 

{1},{3}, or {1,3} 

To, 1^2 

e,w[l\ 

{2} 

ToWl 

e,w\2\w\l\ 

{1,2}{2,3} or {1,2,3} 

To 

e 


Proof. Direct calculation. □ 

Remark 3.5. As elements of GSO 12 , the double coset representatives are identified 
with permutations 0 /{1,..., 12}. Writing these permutations in cycle notation, we 
have tc[l] = (1,2)(11,12), t(;[2]t(;[l] = (1,3,2)(10,11,12). Replacing tc[l] by tc[3] 
in any of the representatives above produces a different element of the same double 
coset. 

3.2. Proof of theorem l3.1L We now apply this description of P\GIRq, to the study 
of Hfx-,s , (p,(/)). For this section only, let wq be the permutation matrix attached to 
(HU), and let w be an arbitrary representative for P(F)\G{F)/Rq(F). 

The global integral ([7]) is equal to 

w^P(F)\G{F)IRq{F) 
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where 




•J ^ { 


IZ{A)U^(F)C^{F)\Cq(A)Uq{A) 

where Cq = CqH w~^Pw, and Uq = Uq n w~^Pw 


f^.s{wug)e{(j)J{ug))Lp{g)dg, 


Proposition 3.6. Ifw does not lie in the double eoset eontaining wq, then Iw{fx;s, ¥^ 2 ) 
0. Consequently, = Iwo{fx-,s,V^i,^ 2 )- 


Proof. Write w = au where tt; is a permutation of 12} satisfying the four con¬ 

ditions listed at the beginning of section 13.11 and u is one of the representatives for 
Pcr\MQjCQ given in the table in lemma [331 The integral (p, 0) vanishes if 

ipi is nontrivial on Uq := Uq n w~^Pw, or equivalently, if the character v ■ fji ob¬ 
tained by composing fji with conjugation by n is nontrivial on Uq n a~^Pa. For our 
representatives u, we have 




+ Z 2 J 0 ), 
' ipiZi^io + -^ 2 , 9 )) 
+ -^ 3 , 9)5 


z/ = e, 

F = m;[1], 

F = t(;[2]rc[l]. 


There are 25 possibilities for a. However, it’s clear that Iw{fx\s,‘^,4>) vanishes, 
regardless of f, if (t(1) < cr(9), or if a(2) < cr(lO). This eliminates all but seven pos¬ 
sibilities for a. For the remaining seven, the above criterion shows that 
vanishes unless f is trivial. 

Assume now that tjji is trivial on Uq. This means that the image of Uq in the 
Heisenberg group intersects the center trivially, and maps to an isotropic subspace 
of the quotient T-LulZ{T-Ln) (which has the structure of a symplectic vector group). 
Write V for this subspace and V-^ for its perp space. Define Py <= Spi^ as in lemma 
12.21 and let Py denote the kernel of the canonical projection Py GLi x SpfV^ jV). 
It follows immediately from lemma 1^31 and the cuspidality of ip that Iw{fx-s,‘P,4>) 
vanishes whenever Py n Cq contains the unipotent radical of a proper parabolic 
subgroup of Cq. This applies to each of the remaining double coset representatives, 
except for wq. □ 


The following lemma is useful in our calculation. 


Lemma 3.7. Let fi, f 2 be two continuous functions on {F\A)^, and ip a nontrivial 
additive character on F\A. Then 

f dxfi{x)f 2 {x)= Y, f dxfi{x)ij(a-x) f dy f 2 {y)^p~^(a-y). {13) 
J{F\A)^ J(F\A)» 

Moreover, if dx fi{x) = 0, then one can replace Y,a^F^ 

formula above. 
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Proof. By Fourier theory on F\A, 

where fi(a) = dx fi{x)'if{ax) for z = 1,2. So the left hand side of (IT^ is equal 

to 

E /i(«)/2(^) f dx'i/;(-(a + ^)-x). (14) 

The integral on x vanishes when a + /3 ^0, and equals 1 if a + /? = 0, so (fTTD equals 

E /i(«)/2(-«), 

a^F” 

which is the right hand side of ffT5]) . When dx fi{x) = 0, we have /i(0) = 0, 

so we can replace Y^a^F^ by Ya^F^-o ■ 


From now on, let w = tc [64321465432465434654]. Then 



' 

1 


\ 


Uq := UqDW ^Pw = ■ 

UgiVT^Vs) = Uq 

0, 

0 0 

0 0 

, 0 

00 



\ 

0 

0 




(15) 


C^:=CQnw-^Pw = {PixP 2 y 


where Pi is the Klingen parabolic subgroup of GSp^ and P 2 is the Siegel parabolic 
subgroup of GSO^. Let Pi = Mi ik Pi and P 2 = M 2 ix P 2 be their Levi decompositions. 
Note that fx]s{wug) = f^.s(wg) for all u e Uq. So, by proposition 13.61 I(fy..s,ip,(f)) is 
equal to 


L 


Z{A)C^{F)\Cq{A) 


pig) / fx-,siwu 2 g) / 9i(j),jiuiU2g))duidu2dg. (16) 

Ju^{A)\U{A) J[U^] 


But, for u = UQiyT,ys) (dehned in (ITSIB . 

[uj^ijiu))(fi]iO = MO^i^YVr + ^sl/s), ^ = 

for any ^1 € i5(Mat4x2(A)). It follows that (IT^ is equal to 


6\ 

es ^6 

Ur 6/ 


(17) 


L 


Z(A)C^(F)\Cq(A) 




U^{A)\U{A) 


f^.siwug)eoi(j),jiug))dudg, 


(18) 
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where 


9 o{(j),ug) := ^ [uj^{ug) 4 ] 

^6Mat3x2(E) 


.ol ’ 


{u € 'Hi7(A), g e Spiq(A)) 


Now, Cq = (Ml X M2)° X {Ui X U2), and f^.s{wuiU2g) = f^-s{wg), for any ui e Ui, 
U2^U2, and g ^ G. Moreover, if 


U2{a) = 


/I a \ 

1 -a 
1 

V 1/ 


( 19 ) 


/6 6\ 

6 ^4 

6 ^6 

\o 0 / 

lows from the cuspidality of that flTS]) is equal to 


then [u^{U2{a)uig)(f)] 


= '0(a(6^6 - ^4^5))[^^{uig)(p] 


/6 6\ 

^3 ^4 

^5 ^6 

VO 0 / 


. It then fol- 


L 


Z(A)C^(F)\Cq(A) 


V X V , , V X 

where 
9 i{(j),ug) := 

The group (Mi x M2)° is the set of all 


U^(A)\U(A) 


f^.s{wug)9i(4),j(ug))dudg, 


( 20 ) 


^€Mat3x2(E): (?3?6-C4C5)+0 \ } 


{u e ^i7(A), g e S'pi6(A)). 


m{g 3 ,g 4 ,t) := diag{tdetg 3 ,g 3 ,t ^^det g3g4,gl-,tdet g 3 ,det g 3 ■ g 3 ,t (21) 

where g3 e GL2, g4 e GL2 and t e GLi. Note that the summation over (^1,^2) is 
invariant under the action of (Mi x M2)°. Consider the action of (Mi x M2)° on 


{( 6 , 6 ,^ 6 ) I det 


ks 

Us ^6/ 


0 }. It is not hard to see that it is transitive, and 


the stabilizer of (1,0,0,1) is {m(t, g3, g4) \ g4 = g3 • det(173) which is the same 

as {M^{t,g3) = diag{tdeig3,g3,t-^]g3,gldeig3,tdeig3,gldeig3,t-^) : g3 € GL2, t € 
GLi}. We denote this group by M5. Let jAi/j be a character on U2 defined by 
'i/'t/2( 1/2(0)) = '0(®)) then equation fl2U]) is equal to 


L 


Z(A)M5(F)Ui(F)U2(A)\Cq(A) 




(U2,'|/'C/2) 


{ 9 ) f 

Ju 


U™(A)\U(A) 


fx;s{wug)92{j{ug))dudg, ( 22 ) 
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02 ij{ug)) := ^ [uj^{j{ug)).(j)] 


where 

e2\ 

1 0 
0 1 

\0 0 / 

and the notation jg dehned as follows. For any unipotent subgroup V of an 

F-group H, character d of V, and smooth left l/(F)-invariant function $ on H(A), 
we dehne 

f <^(vhW(v)dv. 

Jm 


Now, Ui consists of elements 

Ui{a,b, c) = 

and for any g € Rq, 

[uj^(l/i( 0 , 0 ,c)g)(/)] 


nv] 

{lab c\ 
1 b 
1 -a 


e GSp4, 


(23) 




1 / 


1^1 

6^ 


1^1 6\ 

1 

0 

0 

1 

=[wp(^)0] 

1 0 

0 1 

\0 

0 ) 


lo 0 / 


Factoring the integration over Ui and applying lemma 1X71 to functions 

/6 6 \ 

1 0 
0 1 

\0 0 / 


(a,6) ^u^(Ui{a,b, 0 )g)(j) 


and (a,6)i^ / dc(p 2 {Ui{a,b,c)g), 
Jf\a 


we deduce that fl22D is equal to 

/ 


Z(A)M5(F)Ui(A)U2{A)\Cq(A) 


J \j{ug))dudg, 


U^{A)\U{A) 


(24) 

where i/j^f (Ui(a,b,c)) = 'ijj{aa + ( 3 b), = U 1 U 2 , and = ipu^'^’uf ■ group 

M^{F) acts on 17i(A) and permutes the nontrivial characters transitively. The 
stabilizer of 'tfjui '■= is 


Mq:= {Me{ai,a 2 ,ai)} , where M6(ai,a2,a4) = M5 I a4\ I 


(25) 
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Hence equation (12^ is equal to 

f J f^.,{wug)ef^^'^^\j{ug))dudg, 

Z(A)M6(F)Ui(A)U2(A)\Cq(A) U-(A)\U(A) 


where i/jus = 
Note that 


(26) 


[u^{Ui{a,b,0)g)(j)] 

1 0 

0 1 

= [wp(^)0] 

'”6 + fl ^2 + b'^ 
1 0 

0 1 


lo oj 


1 0 0 / 


and that for .^ 1,^2 ^ • (a + ^ 1 ) + /3 • (& + ^ 2 )) = ■ a + (3 - h). We can combine 

the summation on (.^ 1 ,^ 2 ) with the integral over {a,b). It follows that = 

Io{(p,g), dehned in flTUD . Let Mq = UqTq be the Levi decomposition. It is not hard to 
see that both (?) and the function g 1 -^ f{wg) are invariant on the left by Uq{A). 
So, fl2BD is equal to 


I 


‘^(^4’W4)(^) J f^.,{wug)ef^’'^'^^\j{ug))dudg, (27) 


Z(A)r6(F)Ui(A)!72(A)\CQ(A) 


C/™(A)\f/(A) 


where f/4 = U^Uq, and is the extension of ipus to a character of which is trivial 
on Uq. Now, 

//I \ \ //I r \ \ 




(U4,yt/4) 


= /n* 




1 r 
1 


\ 


1 / 


9i 


,1U2,'^2) 

Y2 




\ 


1 -r 

1/ 


^2 


/ 


Let W denote the standard maximal unipotent subgroup of GSp^^ and W that of 
GSO/^. Let and be the extensions of ipui and 'il^u 2 to characters of Ni(A) 
and N 2 {A) respectively, such that 



(1 

\ 


H r 

\ 



1 r 

1 

< 

II 

1 

1 -r 



\ 



\ 




Then it follows from lemma 1X71 (and the cuspidality of (pi,(p 2 ,) that 

■yeF^ 
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where N = N 1 N 2 , a. maximal unipotent subgroup of Cq, and for 7 e F’', 

We plug this in to (ETj). The group Tq acts on the characters transitively, and the 
stabilizer of i/jn ■= is the center of Cq. Since = W:p(g), this completes 

the proof of theorem I3.11 


4. Preparation for the unramified calculation 

In this section, we establish some results which describe the structure of the sym¬ 
metric algebras of some representations of Sp^ x SL 2 , and Sp^ x SL 2 x SL 2 , which 
will be used to relate our local zeta integrals to products of Langlands L-functions. 

We hrst consider some representations of 5*^4 x SL 2 . Let Wi and tn 2 denote the 
fundamental weights of Sp^ and zu that of SL 2 . Write V(^ni,n 2 -,m) for the irreducible 
Sp 4 X S'L 2 -module with highest weight nizui + n 2 ZU 2 + mw, and let [? 7 ,i,n 2 ;m] denote 
its trace. 


Proposition 4.1. For i,j-,ni,n 2 and m all non-negative integers, let pij(ni,n 2 ',m) 
denote the multiplicity of V(^ni,n 2 -,m) in sym* V(i_o;i) ® sym-^ V(i_o;0)- Then 


Y, Pi,j{ni,n2]m)TlHlH^x^y^ = 

2,^,721,722,771=0 

_ 1 - tit2hx^y^ _ 

(1 - tihx){l - x2)(l - t 2 x‘^){l - tiy){l - hxy){l - t 2 hxy){l - tix‘^y){l - t 2 x‘^y‘^)' 

Proof. We hrst describe symt V (4 0 ;i)- By treating V(i,o;i)- Write V(ni,n 2 ) for the irre¬ 
ducible S'p 4 -module with highest weight nizui + n2'!A72- Then we may regard V(i,o;i) as 
two copies of V(i,o) with the standard torus of SL 2 acting on them by eigenvalues, 
say, g and g~^. Then, using the well known fact that sym'^ V(i_o) = L(i,o), and the 
decomposition of P(m,o) ® fo(n,o) described in |K71] , 

n min(ni,n-ni) £ LfJ £ 

'IVsym"V(i.„;i,= y y y[n-2<,i] = yy[n-2«,iin-2('], 

ni=0 £=0 j=0 £=0 j=0 

whence 

00 00 1 00 

^x"Trsym"V(i,0;i)= ^ = 

n=0 j,k,n=0 ^ ^ j,n=0 

Using [K71] again to compute V(nj) ® fo(m,o) one obtains 


Y [n,j][m,0]Cx'^'^‘^iy^ 

72,772,ji=0 


1 00 

- V [n + m + i 2 - k 2 ,j + . 

1 — tXy ri,i,i2,k^0 

n-\-i2'>k 
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It follows that 


i,j,ni,n2,m=0 

1 1 


E t? 


n\+m+i2-k±n2+k±n\ ni+2n2+2i2^,m+i2+k 

Lr\ Ln JU 


1 - 1 - t^ixy 

Ti.+i 2 

1111 


y 


1 - x2 1 - t^iXy 1 - t2X^ 1 - tiy 

n+i 2 ^fc 


E ^ni+i2-k^k^ni ^ni+2n2+2i2 yi2+k 
2 ^3 ^ ’ 


and the result then follows from the identity 


oo oo 77.2 


^ ^ ^ U"'^v"‘w-^ = 


1 - u^vw 


ni=0n2=0i,fc=0 (1 - «)(1 - «^)(1 - ^W)(l - UVW) ' 


□ 


Corollary 4.2. Forn= (ni,n 2 ,n 3 ) let [n] = [ni, 77 - 2 ;%], and, let 

a = * [101122 2 ] 


9 = 


6 = *[0 1 1 1 0 1 2 ] 

Then Y.Zo Tr sym* V'(i,o;i) Ej°!o 9^ Tr sym^ 1 ^( 1 ,o;o) equals 
1 


1 1 0 0 0 1 0 
0 0 0 1 1 0 1 
10 110 0 0 


1 - x^ 


^[n • £/]x’"-V^ - ^[n ■ ^ + (1,1, l)]x'^-^^^y 


, 72 * 6+2 


where n is summed over row vectors n = (ni,..., nj) e Z>q. 

Our next result describes the decomposition of sym* V(i^o;i)®syni'^ l^(i, 0 ; 0 )®syni^ l^(i, 0 ; 0 )- 
It is an identity of rational functions in 6 variables. To keep the notation short, we 
often reflect dependence only on arguments which will vary. Let 

d{ti,t 2 ) = (1 - tit 3 x){l - t 2 x‘^){l - tiy){l - t 3 xy){l - t 2 t 3 xy){l - tix'^y){l - t 2 X^y‘^) 


T-l 




where gi,g 2 and g^ are the three columns of the matrix g in corollary 14.21 Define 
rational functions 71,... ,77 by 

Vl N 2 ni+n 2 

Y Y Y = 7i(m, V, w) + 72(m, V, w)u^^ + 73(n, v, w)v^'^ + 74(7/, v, w)u^^v^'^ 


ni=0n2=0 k =0 


+ J5{u,v,w)(uw)^^ + Jq{u,V,w){vw)^^ + 'yj{u,v,w){uw)^^{vw)^'^ , 
and let c* = '^i{tilz,tizlt 2 ,t 2 zlti). 
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Proposition 4.3. Let denote the multiplicity of V(^ni,n 2 -,m) sym* 

sym-?' V'(i,o;o) ® sym'= V'(i,o; 0 )- Then 


oo 

X! Hi,j,k{ni,n2-,m)tfH'fHfx"y^z^ = (1 - (1 - tiz)~^x 

2 ,j',A:,ni,n2,m=0 

^ Ciu{t2z) ^ C2P(tit2) C3Z/(tiZ^) C4P(t^2:) C^vjtlz) ^ Ceiy{t2Z^) C7v{tlz^) 

d{z,t2) d{ti,t2) d{z,tiz) d{ti,tiZ,) d{t2Z,t2) d{z,t2Z^) d{t2Z,t2Z^y 

where ci,..., C 7 and d are as above and let n(u) = 1 - ut^x^y'^. 

Proof. From [K71] again, one deduces that 


00 


[mi,m 2 ] ■ 

^-0 


00 mi m2 ^ 1+^2 

X! X! X! YXi'^ ^2 - k + ^,m2- i2 + 


^=0ii=0i2=0 k^O 


Combining with corollary 14.21 gives 


X! fii,j,k(ni,n2;m)tytyt^xYz^ = 


i,J,k,721,722,m=0 

1 1 
1 - X^ 1 - tiZ 


E 

neZlQ 


xU'ayn-bf 


n-gs 

3 


(n-gi n-g 2 ii+i 2 

EE E ^n+*2-fc^’^-92-*2+fc^n-gi-ii+*2+fc 

* 1=0 * 2=0 fc =0 


-x^y% 


n-gi+l **- 32+1 *i+*2 

E E E jii-\-i2-kj^2-g2-^^-‘i'2'^k^rL-gi-\-l-ii+i2+k 

ii=0 22^0 k^O 


7 


and and simplifying this rational function gives the result. □ 

Proposition 4.4. Let V’(r*i,n 2 ;r* 3 ;r* 4 ) denote the irreducible representation of SpixSL 2 x 
SL 2 such that Spi acts with highest weight niW\+n 2 W 2 , the first SL 2 acts with highest 
weight ns, and the second SL 2 acts with highest weight n^. For n = (ni,n 2 ;ns; n 4 ), 
let denote the multiplicity ofVn in sym* P(i^o;i; 0 ) ® sym-? V(i^o; 0 ;i)- Then 


E E PiA^yytyqHyx^y^ = 

*.1=0 r,cy 4 


^(x,y,t) 
6{x,y,t) ’ 


'>0 
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where 

V 


(x, y, t') =1 - tit2t‘j,t\x^y^ - t]t2tltiX^y^ - - t\t2t‘itiX^y^ - tit2tlt4X^y^ 

- tit2ht\x‘^y^ - At2tlx‘^y'^ - t\t2t\x‘^y'^ + 2t\t2tlt\x‘^y'^ - tltltlx^y^ 

+ t\t2t^t\x^y‘^ + tit\t\t\x^y^ + t\t2tlt/^x‘^y^ + tit\tlt\x‘^y^ + t\t2t\tiX^y^ 

+ t\tlt\t4^x^y^ + t\t2t^t\x^y^ + t\t2t^t\x^y^ + t\t2t\t^x^y^ + tit\tlt\x^y^ 

+ t\t2t2,tlx^y^ + tity:\t\x^y^ - t\t2tltlx^y^ + 2t\tltlt\x^y^ - t\t2t\t\x^y^ 

_ 6,,6 _ ^ 3 ^ 2 ^ 3^2 7„,6 _ . 3 ^ 2 ^ 2 ^ 3 „ 6,,7 _ . 2 . 2 ^ 3 ^ 3 ™ 7,,7 

- tltltltlx'^y'^ - tltltjtlxW - tltltlt^y^ + tttl4ttx^°y^°, 

S{x,y,t) =(1 - tit 3 x){l - x^)(l - t2x4{l - tiUy){l - y^i^ - i 2 ?/^)(l - x^y^i^ - t^Uxy) 
(1 - t2ht^xy){l - titiX^y){l - iit3X?/^)(l - tlx^y^i} - t2tlx^y4{^ - t2tlx^y4 


Proof. Let p and q be the polynomials such that 

^ hijAnun2,m{ni,n2]rri)tfH2Hf'x''y^z^ = ■ 

i,j,fc=o Q{x,y,z,t) 

They may be computed explicitly using proposition 14.31 Set f = (ti,t 2 ,^ 3 ), and 


f{x,y,t',U) = Y, E hiqir^tfH'fHlHl^x^yP 

ij =0 neZ^^ 

By regarding V(i,o; 0 ;i) as two copies of V(i^o; 0 ) with the standard torus of the second 
SL 2 acting with eigenvalues r and r"^, say, we see that 

xfjx, y, t', r) - r-V(x, y, t', r"^) ^ ^ ^ p{x,yT, 7/r-^f) 

(r-r-i) ^ q{x,yT,yT-^,t')' 

So it suffices to verify that 

p{x, yT,yT-\t'){T - T-^hix, y, t', t)6{x, y, t', r"^) 

= Qix, yr, yT~\t')[Ti^{x, y, t', t)5{x, y, t', r"^) - r“V(x, y, f, t~^)S{x, y, t', r)], 
which is easily done with a computer algebra system. □ 


5. Local zeta integrals I 

5.1. Definitions and notation. The next step in the analysis of our global integral 
is to study the corresponding local zeta integrals. We introduce a “local” notation 
which will be used throughout sections [5] and [HI [3 In these section F is a local held 
which may be archimedean or nonarchimedean. Abusing notation, we denote the 
F-points of an F-algebraic group F by F as well. We hx an additive character if 
of F, and dehne a character ifj^j : F -> C by the same formula used in the global 
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setting. Similarly, if we fix a triple y = {X 11 X 21 X 3 ) of characters of F"", and s e C^, 
then formula ([2]) now defines a character of Mp. We write Indp(x;s) for the corre¬ 
sponding (unnormalized) induced representation (JWfinite vectors, relative to some 
fixed maximal compact K). We shall assume that the characters in x are unitary, 
but not that they are normalized, and define (y; s) for s € by the convention 
S 3 = have a two parameter family of induced representations and let 

Flat(y) denote the space of flat sections. 

Let i 5 (Mat 4 x 2 ) be the Bruhat-Schwartz space, which we equip with an action 
of Rq as in the global setting, and define Iq : i 5 (Mat 4 x 2 ) x Rq ^ C by replacing A by 
F in flTU]) . 

Next, take tt to be a "^Ar-generic irreducible admissible representation of Cq with 
■ 0 AT-Whittaker model and with central character Xi^X 2 ^X 3 - 

For W e € Flat(x) and (p e i 5 (Mat 4 x 2 ), define the corresponding local 

zeta integral to be the local analogue of flT^ . namely: 


f W(g) f f^.s{wug)Io{(p,ug)dudg. 
JZUi\CQ JU^Wa 


u^\Uq 


(28) 


In addition to the above notation, for 1 < i,j < r,i j, let Xij{r) = I 12 + rEij - 
where lu is the 12 x 12 identity matrix and Eij is the matrix with a one 
at the i,j entry and zeros everywhere else, and let 


/O 0\ 
1 0 
0 1 

\o 0 / 



0 - 2 '^ 

0 

04 

0 

0 

lo 

oj 


(a = (oi, 02 , 04 ) € F^). 


5.2. Inital computations. In this section we carry out some initial computations 
with local zeta integrals which will be used in both the proof of convergence in section 
16.11 and in the unramihed computations carried out in section O 

The image of the function 0:23 maps isomorphically onto the one dimensional quo¬ 
tient of Ui\N, and the function g 1 -^ fs{wg) is invariant by the image of 0:23 on the 
left. Moreover, W{x 23 {xi)g) = 'ilj{xi)W{g), while 


[oj^ {x 23 {xA)ug) 4] (So + S 2 (xi,X 2 , 0 )) = [u^ (ut) 4] (So + S 2 (Ti,a: 2 ,X 4 )) 
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Let 



L 2 \ 

L4 

1 

0 / 


' 0 (ri + r 4 ) dr, 


{(j) e 5(Mat4x2)) 


h 


E{f,(j),s):= f f^.s{wu)M{uj^^i{u).(j))du, (^ € 5 (Mat 4 x 2 ),/e Flat(x)) 

JU^\Uq 


where 0 e i5(Mat4x2),/ ^ Flat(x))^ ^ R is right translation. Then 

expressing Haar measure on Cq in terms of Haar measures on T, N and K, and 
using X 23 to parametrize U^\N yields 

liW, f, (t>-.s) = h{R{k).W, R{k).f,ik).<p- s) dk (29) 

where K is the maximal compact. 

The integral iZf(0) is absolutely convergent. Indeed, iZf(0) = ^i(To), where 0i is 
the Schwartz function obtained by taking Fourier transform of cj) in three of the eight 
variables, and Tq is a matrix with entries 0 and 1. We study the dependence on ti € 
Uq\Uq and t € Z\T using the local analogues of (jS]). A remark is in order, regarding 
the Weil index 7 i/,,detm^(gi,c/ 2 ) which appears in the third formula. In order to reconcile 
the local and global cases, one should think of this as the ratio 7 ^,detm^(gi,g 2 )/ 7 h:i- 
denominator can be omitted because the global 7 ^^i is trivial. In the local setting 
7^4 may not be trivial, but 7 ^^a 2 = for any a, and detmQ{gi,g 2 ) is always a 
square, so the ratio is always trivial. 

Now, let Uq c Uq be the subgroup corresponding to the variables, xi,X2,x^, 2 / 3 , ?/ 5 , y&, 
zi,Z 2 ,Z 3 and z^. That is, the subset in which all other variables equal zero. Let 
Uj c Uq be the subgroup dehned by the condition that each variable listed above is 
0, and, in addition, yi = ys = 0. Then UqUj maps isomorphically onto the quotient 
Uq\Uq. We parametrize Uq and U 7 using the coordinates inherited from Uq. A direct 
computation using ([S]) shows that for uq € Uo,U 7 e U 7 , and t € T, M{u^{tuoU 7 ).(f)) is 
equal to 


+X4t^^-y3t ^^+yU + zi +Z7,)(l)' 


' yi + 

X 3 + 

X 5 


\ X 7 


y2 ^ 
2/4 + 

Xq + 


, (30) 


a^8 / 


where (f)' is the Schwartz function obtained by taking the Fourier transform of cj) 
(relative to xp) in Xi,X 2 and X 4 . If we dehne 4’UoA'^o) + X 4 t^'*‘ - y^UP^ + 
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yet ^6 + zi + 2 : 5 ), and we define 6 {t) e Uj and : U'^ ^ Mat 4 x 2 by 


^(t) =uq 



( 0 

0 \ 


1 ° 

0 \ 




0 


0 

0 

,0 


0 

t-Je 

1 


0 


1 0 

0 j 


lo 




^Ur, 


717 ( 117 ) 



92^ 

X 3 

yi 

X 5 

Xq 

yX7 

Xg) 


then flSUD becomes 

The projection 777 has a two dimensional kernel corresponding to the variables 
24 and zq. Let Us denote this kernel and choose a subset t/g of Uy which con¬ 
tains 5{t) and maps isomorphically onto the quotient. Then we can parametrize 
E(R{t).f,u^^i(t).(j),s) as a triple integral over UqxUs^ U'^. The U'^ integral is con¬ 
vergent because cp' is Schwartz, after a change of variables it becomes 


(P' =^1 /x;s(7TtioM8^(t)) := f^.s{wUoUs6{t)u'^)(p'{Tlj{u'^))dUs. 


Thus, conjugating t from right to left, and making a change of variables yields 
hiW, /, (p; s) = UiW, cp =(-1 /; s), where /) dehned as 

f det s){wtw~^)fy-s{wuoUs 6 {t)) dt, 

Jz\T 


with Jaci(f) being the “Jacobian” of the change of variables uo -> tuot~^, U 7 iujU^. 
Notice that (p *i f is simply another smooth section of the same family of induced 
representations, and that if fy.-s and (p are both unramihed, then <p' = cp and (p'*ifx;s = 
fx;s- Thus, we may dispense with the integral over Wg. 

Next, we dispense with the integral over rtg. To do this, we use |DM78j to replace 
fx\s by a sum of sections of the form 

(p2*2fx-s(9)-= fx-A9X24{yi)x34iy2))Myi^y2)dy, (seC^^€G'). 

J 

Now, let 

[E2-fxAi9) ■= / fxA'^'^o9)'ipUo,tMduo, UQ{yi,y2) ■= X24{yi)x3i(y2). 

JUq 

conjugating wg( 7 / 1 , 7 / 2 ) from right to left shows that 


[dd2-fx;s](d{t)u8Ug{yi,y2)) = 'lp{-yiZQ - 7 / 2(^4 - U^)) ■ [E2-fx-,s\{d{t)us). 


(Recall that zq and 2:4 are coordinates on Us-) But then 

f [E 2 -(p 2 *2 fx-,s]{5{t)us) dz = f [E 2 -fx-,s]{d{t)u 8 )'$ 2 {ze, Z 4 - dz, 

J J F^ 

which we may write as [E 2 -(p 2 =^3 fxAAi'^))^ where =(-3 is the action of S{Us) by 
convolution, and 6 (t) = 6 (t)x 2 g(U^). Notice that (p 2 fx-,s is again another smooth 
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section of the same family of induced representations. Note also that if / is spherical 
then taking 02 to be the characteristic function of gives 02 /x;s = /x;^- 
Thus, we are reduced to the study of the integral 

h{WJ-s)-.= f W{t) 5 ]^l\t)usm 2 -fxMt))dt, (31) 

J Z\T 


where Ps(t) := det tja |t/3i+/^2+/34| Jaci(t)(x; s)(wtw~^). Write w = W 1 W 2 W 3 , where 

wi = tc[634], W 2 = tc[3236514] and = re[2356243564]. Write U for the unipotent 
radical of our standard Borel of G, and U~ for the unipotent radical of the opposite 
Borel. For tc € VF, let [ 4 , = [/ n w~^U~w. Then = Uyj^yj^ = W 2 ^Uyj^W 2 Uw^. 

For c := (ci,... jCg) e F®, dehne a character 0c ,0 of Uq in terms of the standard co¬ 
ordinates on Uq by 0c, 0 (^ 0 ) •= 0(ciXi + C 2 X 4 + c^i/q - + c^Zi + CqZ^). Notice that 

0 ; 7 p^t is obtained by taking c = ,t~^^,1,1). In terms of the entries Uij we 

have 0 c,o(m) = 0 (ciMi 5 + C 2 U 26 + c^U 27 - C4ti38 + cs^ig + cqU 2 ,io), Now, W 3 Corresponds 
to the permutation (2,4,11,9)(3, 8 ,10,5). It follows that u'q 4 ’c,o{w^^UqW 3 ) is the 
character of wsUow^^ given hy 0 (citti 3 + C 2 M 46 + 03^47 + 04^35 + C 5 M 12 + cqu^^). In 
particular, its restriction to W 2 ^UwiW 2 is trivial. Let 0c ,2 denote the restriction to 
Fiuj. Then 


/ fxA^'^ 9 )i’c,o(u)du= / / /x;0t«lMlJr'2M2J4'3fi')(ittl0c,2(M2)dW2, (32) 

Uq -J Uwq^ *■' Uw-^ 


and the Ui integral is a standard intertwining operator, M(w^^,x', s) : Indp(x;s) 

IndB^((x;s)(5]j^j^)"^i, where ((x; := {{x', s)S~j^'^){witwi^), and Ind denotes 

normalized induction. 

Now let Wi = zi;[32365] so that W 2 = tC4tc[14]. Also, let w'^ = zi;[14]tC3. Observe 
that W 4 is the long element of the Weyl group of a standard Levi subgroup of GS'Oi 2 
which is isomorphic to GLi x GL^ x GSO^. For ci,..., C 4 e F, dehne a character 0c ,4 of 

f 4,4 by ' 0 c, 4 (m) = 0(ciM23 + C 4 M 34 + C3M56 + C 2 M 57 ), and for /x 
let 

‘^V’c,4 ■/x;s;'!4’l ( 5 ') “ / /x;s,toi (^4Ji5')'0c,4(ji) dti, 

'•'Uw^ 

which is a Jacquet integral for this Levi subgroup. Then (1521) equals 

/ Jb4e,4 ° M{wi^,x; s)-/x;^](^ 2 i(o)a:^ 54 (fo)«^ 3 ^) 0 (c 5 ri + CQr 2 ) dr. 

J 


6. Unramieied Calculation 


We keep the notation from section 15.11 and assume further that F is nonar- 
chimedean, with ring of integers 0 having unique maximal ideal p. We hx a gen¬ 
erator to for p. The absolute value on F is denoted | | and normalized so that 
|tD| = (!'■= #o/p. The corresponding p-adic valuation is denoted v. Moreover, we 
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assume that K = G(o), and that the representation tt and characters Xi, i = 1 , 2,3 
are unramihed, and we let W°,f° and (j)° denote the normalized spherical elements 
of (tt), Flat(x) and i 5 (Mat 4 x 2 ), respectively. 

The (hnite Galois form of the) L-group of GSp^xGSO^ is GSpin5{C)xGSpin4(C). 
Indeed, one may dehne GSpin2n+i (resp. GSpin2n) as the reductive group with root 
datum dual to that of GSp2n (resp. GS 02 n)- However, both GSpin groups appearing 
here can be understood more explicitly via “coincidences of low rank.” Indeed, a 
simple change of Z-basis reveals that the root datum of GSp^^ is in fact self dual. 
Thus GSpin^ is just GSp4 in another guise. Note, however, that the isomorphism 
of GSp4 with its own dual group does not respect the standard numbering of the 
simple roots. 

Next, we can realize GSO4 (resp. GSpin4) as a quotient (resp. subgroup) of GL2 x 
GL2. Indeed, we can realize GSO4 as the similitude group of the four dimensional 
quadratic space (Mat2x2, det). Letting GL2 x GL2 act by (gi,g2) = giX^g2 induces 

a surjection GL2 x GL2 GSO4 with kernel {(a/2,a"^/2) : a e GLi), and thence 
a bijection between representations of GSO4 and pairs of representations of GL2 
with the same central character. By duality, this induces an isomorphism of GSpin4 
with {{gi,g2) £ GL2 X GL2 '■ det^fi = det5f2}- We remark that the induced map 
GSpin4 SO4 is not the restriction of our chosen map GL2 x GL2 GSO4. 

We regard GSp4 x GSO4 as a subgroup of Mq containing Gq in the obvious way, 
and regard its L group as a subgroup of GSp4xGL2^GL2. We make the identihcation 
in such a way that the hrst GL2 corresponds to the hfth simple root of G = GSO12 
and the second GL2 corresponds to the sixth simple root of G. 

Let Stcspi denote the standard representation of GSp4. It may also be regarded 
as the spin representation of GSpin^. For this reason, the associated L function is 
often called the “Spinor L function.” We regard Stcspi as a representation of of 
the L group via projection onto the GSp4{fC) factor, and let denote the dual 

representation. Let S'G.ci) (resp. St^^(2)) denote the representations of the L group 
obtained by composing the standard representation of GL2 with projection onto the 
hrst (resp. second) GL2(C) factor. 


Theorem 6.1. Let 

N(s, x) =L(si - S2, —)L(si - S2 - 1, —)h(si -S2-2, —)L{si + S2-2, X1X2) 

X2 X2 X2 ( 33 ) 

X L(si + S2 - 3 , XiX2)L{si + S2 - 4 , XiX 2 )L{ 2 s 2 , xl)L{ 2 si - 6 , xf)- 


(Local L functions. The corresponding product of global zeta functions is the normal¬ 
izing factor of the Eisenstein series.) Then I(Wf,f°,(f)°;s) equals 


L{ 


S1-S2 

2 


Sl+.g2 




N{s,x) 




GL\ 


X3 \ 
X1X2 t 
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Reduction of the general case to the special case of trivial characters. For purposes of 
this proof, write Xh for the similitude rational character of H where H = GSO 12, GSp^., 
or GSOi- Our embedding (GSp^ x GSO^Y ^ GSO12 is such that XgsOi 2 { 9 ^^) = 
XGSp4,(,g)~^ = XGsOiih), and the projection p : GL2 x GL2 GSO^ is such that 
>^gsoM 9 i, 92 )) =det^i det^fa- 

Write TT = n ® Ti ® Ta where If is an unramihed representation of GSp4^ and Ti 
and Ta are unramihed representations of GL2 with the same central character (so 
that Ti ® Ta is a representation of GSOi). Write r* = Ti^ ® |det|*i where Ti^ is 
an unramihed representation of GL2 with trivial central character for i = 1,2, and 
ti is a complex number, and write fl = flo ® |Agsp 4|*^, where flo is an unramihed 
representation of GSp4 with trivial central character and ta is a complex number. 
Then, as representations of Gq = (GSp^ x GS 04 )°, 

TT = TTo ® |AGsOl2^^”^^ where tto = IIo ® ri,o ® ra,o- 

The operation of twisting flo by \XGSpi\^^ to obtain fl corresponds, on the L group 
side, to multiplying the corresponding semisimple conjugacy class in GSp4{fC) by 
Likewise, the operation of twisting Ti^ by jdetpi corresponds to multiplying 
by in GLa(C) for t = 1 , 2 . If r; is the unramihed character 77(a) = |a|^, then 

L (u, TT, ® X 77) = L (m - ta + ti + l, ttq, ® St^^o)^ . 

For i = 1 , 2,3 the unramihed character Xi is given by | for some r* € C. If 
s = (si, Sa, j ^ then let s' = (si + Li, Sa + ra, 

triple consisting of three copies of the trivial character. Then it follows directly from 

^ , r, o I I 3r-] J-Sr^ 

the dehnitions that = ff-^ s'' |AGSOi2r^ ^ • The general case now follows from 

the case ti = ta = ri = ra = ra = 0. 

Recall that I{W, f, 0 ; s) is only dehned when the triple x and the central character 
of TT are compatible. In the present notation the compatibility condition is that 
-3ri - 3ra + 2r3 + 2(ti - ta) = 0 . But then BAr • f^-s = ^-n-o' fxo s" general case now 
reduces to the special case when x = Xo and % = tiq. □ 

Remark 6 . 2 . IFe may now assume that tt = Il^ri ®ra, where the central characters 
of n,ri and Ta are all trivial. Thus, vr may be regarded as an unramified repre¬ 
sentation of SO5 X PGL2 X PGL2 and corresponds to a semisimple conjugacy class 
in Sp4{C) X SL2{C) X SL2{C). Define Rtsp^,S't^^(i) and Stgj^(2) as the restrictions 
of StGSpi, Stand 5'tg^(2), respectively. Note that all three are self dual repre¬ 
sentations. In particular, L(u,'ir, St^g^^ ® S't^^(i)) = L{u, 7 r, Stsp^ ® 
ti € C. 

Proof in the special case of trivial characters. As x is trivial we write /° instead of 
Let I{s, tt) := I{Wf, f° s). As we’ve seen in section it is equal to l3{Wf,f°] s), 





TWISTED SPINOR L-FUNCTION 


27 


defined by (El])- Let Ii(s; Ci,..., Cg) equal 


/ fxA'^UQ)'lpcfl{UQ)dUQ. 
JUq 


Let a = Note that /3i-/33 = ai, = as, and = 02. 

Moreover, the characters -fd^ - (d^ and ag are not identical on T^-, but they have the 
same restriction to the maximal torus T of Cq. It follows that for t in the support 
of W°, the quantities |ad|, |M|, |6c| and |cd| are all < 1 . 

By plugging in the Iwasawa decomposition for w^ 5 {t), we find that 


E2.f:(6it)) = - 


Ii(s;a,6,c,(i,l,l), 

ad, bd, cd, 1 , 1 , 0 ), 
a, be, 1 , cd, 1 , 0 ), 

|Q|-si+s2+2|c|-si-s2+4j^(-_g^ 1, be, 1, aed, 0,0), 
|q|-si+s2+2j^('5^ 1 , 6 , c, ad, 0 , 1 ), 
|Q|-si+s2+2|5|-si+s2+2j^^_g^ 1,1, be, abd,0,0), 
|^|-si+s2+2];^(^g^ a, 1 , 6 c, bd, 1 , 0 ), 


|a|,|6|,|c|,|d| < 1, 

|d| > l,|a|,|6|,|c| < 1, 

|c| > l,|a|,|6|,|d| < 1, 

|a|, |c| > 1 , |6|, |d| < 1 , ( 34 ) 
|6|, |c|, |d| < 1, |a| > 1, 

|c|, |d| < 1, |a|, |6| > 1, 

|d|, |c|, |a| < 1, |6| > 1. 


Now let denote the normalized spherical vector in Ind p„((Yn: g)d^^)"^b and let 

V ( ) — C(2g2 ~ 1) C(-Sl ~ ^2 ~ 1) C('Sl + -§2 ~ 3) 

C(2S2) C('Si ~ S 2 ) C('51 + 'S2~2) 

Then, M(w^^,x', s).f° = ■Z'i(s)/°^^, by the Gindikin-Karpalevic formula, and hence 

Il(5,Ci, . . . , Cg) — Z\ ( 5 ) / 

, 4 :fs,Wi (a:2i(ri)a:54(r2))^(c5ri + Cgr2) dr. 

J 

_ 1 

We remark that ((xo;5)6^^maps diag(Afi,..., Afg ,,... ^ Tq to 




( 35 ) 


Lemma 6 . 3 . Assume that each of e^,eQ is either zero or a unit. Assume further that 
if Cq = 0 then at least one of 02,03,04 is a unit, and that if 05 = 0 then ci is a unit, 
and set Jci,c2,c3,c4 = Ji^cAfs,wiih2)- Then Ii(s;ci,... ,cg)/Zi(s) equals 


J, 


^-si 


Cl,C2,C3,C4 


-S2+4j^ 


,C2,C3,C4 




’ ro ’ tn ’ ro 


g-Ssi-S2 + 10 


£2 £2. £4 , 
itj ’ ttj ’ itj ’ ro 


Remark 6 . 4 . Observe that all the sextuples Ci,...,Cg appearing in (l 34 l) satisfy the 
conditions of lemma Id.dl 


Proof. There exist cocharacters hi : GLi Tq, {i = 1 , 2 ) such that (6,1, a*) = and 
(6-2,a*) = dj_4 (Kronecker 6). It follows that J7v'c.4/s%i(^2i(i’i)a^54(?"2)) depends only 
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on v{ri) and v{r2)- If C5 is a unit, then 

f 'ilj{c 5 ri)dri = \ ^ I’ 

Jv(ri)=-k [^0, A; > 1, 

and similarly with r2. Since both and ^|J are unramified, it follows that Ii(s; Ci,..., 
equals 

Plugging in the Iwasawa decomposition of X2i(rD~^) and/or X54(tt)”^) gives the result 
in this case. 

Now suppose that C5 is not a unit. Then it is zero and ci is a unit. It follows that 

/ , Ji^o,Jln,^{x■ 2 l(rl)x 54 {r 2 ))^p{c 5 rl + Cer 2 )dr = / j 7 pe, 4 /,%,(x 54 (r 2 ))V^(c 6 r 2 ) dr2. 

J J F 

Indeed the support of J is contained in UTiK where K = GSOi2{o) is the maximal 
compact subgroup, and Ti is the set of torus elements t with |t“2| < 1 . It follows easily 
from the Iwasawa decomposition that X2i(ri)a;54(r2) e UTiK if and only if ri e 0. If 
Cq is a unit then proceeding as before we obtain 

/ Ji^cAflwiix 54 i,r 2 ))^{cer 2 ) dr 2 = Jci,c 2 ,c 3 ,c 4 " 

J jp -r , , ± ro ? ro ’ ro 

On the other hand, when ci is a unit then = Jq £2 £3 £4 = 0 , and the stated 

^ ,C 2 ,C 3 ,C 4 ^ ^ 

result follows in this case as well. Likewise, if cq is zero, then integration over r2 can 
be omitted and 22. 22. 2± = J21 22 22. 22. =0, which gives the result in the remaining 
two cases. □ 


Now consider the subgroup of the torus consisting of all elements of the form 

t = diag{tit2,t2,l,ti^^,tit2,t2,l,ti^)- (36) 

This subgroup maps isomorphically onto Z\T. For elements of this torus and with 
coordinates as in fl56D we have 


Jaci(t) = Itjtstl 


Jac 2 (f) = 




tstl 


V = {tihhUl 


-1 


|dett |2 = \t 2 ^tlt^' 


{Xo]s){wtw "^\t 2 


Sl-S2p„|-Sl-3s2+S3|^g^2|S2 


^ ^ S-] —3s-;) ^ ^ ^ _ 

Set X = q~^ 2 L y = and let Ui be the p = v^U) for 1 < i < 4. Then 


Us{t) = a;2ni+n2^2ni+n3+2n4^ (- 37 ) 

For I = (/i, l2,h, h) e set 

= 2 — ^2^4+2^2/4+2 _ y2li+2 _ ^ 2/1 +2/4+4^4/i+4/4+8 _|_ ^2/4+2^2/i+4/4+6 _|_ ^2/i+2/4+4^4/i+2/4+6 
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_ 72(0 = 1 ~ ^2l2+2yAl2+i ^ 3(0 = 1 - x^^3+2^2Z3+2 

■n\ ^ |ii(0i2(/)i3(0, >ovi, 

|o, otherwise. 

Then direct computation or the Casselman-Shalika formula shows that 

^ fo (T \ _ C(gi + S2 - 4)^ C(gi - g2 - 2)^ C(252 - 2) . . 

12 J 3^2 _ 1)2 ^ 2 s 2 - 1)-^^ 

where li = v{ci) for each i. Hence, if 

h (0 = i (0 ( 1 > 0 , 0 , 0 )) ( 0 , 1 , 1 ,!)) + ( 1 , 1 , 1 ,!)), 


then 


Ii(s; Cl,...,Ce) - 


C(si - S 2 - 2)2C(si + S 2 - 4)2C(2s 2 “ 2)H(ril, n 2 , Rs, R 4 ) 


(((si - S 2 )C('Sl - S 2 ~ l)C('Sl + S 2 - 2)C(si + S 2 - 3)C(2S2) 
for all Cl,... ,C6 satisfying the conditions of lemma lOl Consequently, 


jr C(gi - ^2 - 2 )^C(gi + ^2 - 4 )^C( 2 g 2 - 2 )^(kl, H, C, D) 

2-Js {{))- ^ _ 2 )^( 5 ^ ^ _ 3 )^ 2 S 2 ) ’ 

where A = 'i'(t^i), B = C = n(C^®), and D = and i is dehned piecewise 

in terms of ii according to the seven cases from (j34|) . It is convenient to use an 
alternate parametrization. Let ^(mi,m2,m3,m4) equal 


z(mi-- 


-m2 + m3 -I- m 4 m2 -I- m3 - m4 m2 - m3 + m4 -m2 -1- m3 -1- m4 


) ^2m\ +m2 y2mi+m2 +m 4 


if m2 -I- m3 -I- m4 is even and zero otherwise. Then 

C(si - S2 - 2 ) 2 C(si -I- S2 - 4)2C(2s2 - 2 )ii{m) 


(38) 




C('5l ~ 'S2)C('Sl - S 2 - l)C(si -I- S 2 - 2)C(si -I- S 2 - 3)C(2s2) 
where now m, = n(t“») for i = 1,2, 3 , 4 . Let [mi,m2;m3;m4] or [m] denote the trace 
of the irreducible representation of ^{CqIZ) := Sp^ x SL2 x SL2 on which Sp4 acts 
with highest weight mitui +m2ZU2, the hrst SL2 acts with highest weight m3, and 
the second SL2 acts with highest weight m4. Then 

_ 1 

= [m 2 ,mi;m 3 ;m 4 ](r^), 


where is the semisimple conjugacy class in ^(CqIZ) attached to tt. Note the 
reversal of order between 1 and 2 . The reason for this is that when GSp^ is identihed 
with its own dual group, the standard numberings for the two dual G5'p4’s are 
opposite to one another. For example the coroot attached to the short simple root 
ai is the long simple coroot, which makes it the long simple root of the dual group. 

Now, let Z2{x,y) = (1 - y‘^){l - x‘^y'^y{l - x'^y^y. Then j(n) is divisible by Z for 
any n. Also, for g the character of a hnite dimensional representations of ^{CqIZ), 
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let L{u, g) = Then theorem EH] is reduced to the following identity 

of power series over representation ring of ^{Cq!Z) : 

—^—- y ii{m2,mi,m3,m4)[m]= Z3{x,y)L{xy,[l,0-,l]0])L{xy‘^,[l,0]0-,l]), 

( 39 ) 

where ^3(x, y) = (1 - - x‘^y‘^){l - x^y^). 

Dehne polynomials Pm{u,v) by 

L(w,[l,0;l;0])L(n,[l,0;0;l]) = ^ Pmiu,v)[m]. 

meZ^Q 

Then ( 13 ^ is equivalent to the family of identities of polynomials, 

(1 - y‘^)(l - x'^y'^)^{l - - x^y^)Pm{xy,xy‘^) = ii{m 2 ,1711,1713,1714) (Vm € Z^q), 


or to the identity of power series over a polynomial ring: 


^ ii{m2,i7ii,m3,m4)tyt^^t'^^ty 

meZ^Q 


^4(x,y) X, ^m(xy,xy^)tyt^^t^H', 

meZ^Q 

p(x,y,t) 


7714 

4 


Z4(x,y) 


S(x,y,t) 


( 40 ) 


where p and S are defined as in proposition 14.41 and Z4 = Z2Z3. 

The identity ( 140 ]) can be proved as follows. Let X = (Xi, X 2 ,X 3 , X4) and V = 
(Ti, 12, ^4) be quadruples of indeterminates. Define polynomials, 


ijx, y, X, Y) := l-xYXlYi-y^Y 4 ^-x^y^XfXlY 4 ^Y,Ux^y^XlY 3 ^Yi+x^y^XlXlY,^Yi 
l 2 {^,y,X,Y) = {1 - x^y^X^Y^)] i^{x,y,X,Y) = (1 - x’^y^XlY^); 1 = 141213 , 


so that for k = (ki ,..., /C4) e Z>q, the polynomial j(k) is equal to j(a:, y, x^,y^), where 
xk (x*^i,... ,a:^'‘) and y^ := {y^^,... ,y'^^). Likewise, one computes a polynomial 
i4(x,y,X,Y) such that ii{k) = i3{x,y,x^,y^). It can be expressed as a sum of 12 
monomials in X and Y, each with a coefficient which is a polynomial in x and 
y. Thus ii{k) = 2/) j{x,y))^^, for some polynomials ci,...,ci2 and 

monomials ..., 1x42^4 in x and y. Now, 


^ ii{rn2, nil, 1713,1714) 


E 

A,B,C,D€Z 

A+D,B+C,B+D,C+D>0 


i{A, B, C, D)x 


2A+B+C+2D 2A+B+2C+3Dj.B+Cj.A+Dj.B+Dj.C+D 
y I'l ^2 f-S ^4 
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This is a sum of seven subsums corresponding to the seven cases which appear in 
). The simplest of these is 




A,B,C,D>0 
12 


= E 


Ci{x,y) 




In each of the other six sums one can make a substitution to obtain a similar, fourfold 
sum of A', B',C', D' from 0 to inhnity. For example, in the second case listed fl 34 p . 
one has the conditions A,B,C > -D > 1 . Substituting A' = A + D, B' = B + D, 
C = C + D, and D' = -D - 1 , yields 


E h{A',B',C', 0 )tl 

A',B',C’,D'=0 

12 


R'+C"+2D'+2+A'+BOC' 2A'+_B'+C'+4_D'+4,,2A'+B'+2C'+6D'+6 


= E 


ti X 


Ci{x,y)x'^yHl 


y 


(1 - fii,i{x,y)Bx‘^y^)(l - yi^2{x,y)tit3xy){l - fii^3(x,y)tit4xy'^){l -tjx'^y^) 


The other hve subsums are treated similarly. Totaling up the resulting rational 
functions and simplifying gives fl 4 UD . completing the proof of theorem □ 


6.1. Convergence. In this section, we prove the convergence of local zeta integrals 

Theorem 6.5. Take W € Vy^jy(7r),/ € Flat(x) and (j) € i 5 (Mat 4 x 2 )- Then the local 
zeta integral I{W, /, </>; s) converges for Re(si - S2) and Re(s2) both sufficiently large. 

Proof. We need to show that convergence of fiW^f^s) dehned in (ITT]) , for W e 
W 0 jv('^) / a smooth section of the family of induced representations Indp(x; s). 

To do this, we simply bound f^.g by a constant times the spherical section 

where Re(s) € is the real part of s. Then E2-fx;sid{t)) is bounded by a constant 
multiple of M{wfwf ,Re{s)).f^^^^.^(w36(t)), where M(wfwf ,Re{s)) is a standard 
intertwining operator. The unramihed character (xq)R e(s))(Ip|^ may be identihed 
with an element of X{Tg) M. The integral dehning the standard intertwining 
operator converges provided the canonical pairing is positive for all positive 

roots a with wfwfa < 0 . Inspecting this set of roots, one hnds it is convergent 
provided Re(s2) > l,Re(si - S2) > 2 , and Re(si + S2) > 5 . Moreover, it converges to a 
section of the representation induced (via normalized induction) from 

Next, we need to understand the dependence of M{wiW2,Re{s)).ff^^^^^{w36(t)) 

on T. In order to do this, we write W3S{t) as V{t)T{t)'R{t) where V{t) e U, T{t) e Tq 
and ikif) varies in a compact set. It is convenient to do so using the basic algebraic 
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substitution 



( 41 ) 


which corresponds to the Iwasawa decomposition if F is nonarchimedean, but remains 
valid in the Archimedean case as well. 

Recall that 5 {t) is the product of X3q{-c) and X25{-d)xi^{-a)x29{ad), 

which all commute with one another. We can partition T into 16 subsets and use 
the identity im to obtain a uniform expression for r(t) on each subset, and compute 
in each case, obtaining 


f 1, \b\<l, 

I | 6|>1 


X 





|a|,|(i| < 1, ' 

1, |c| < 1, 

... 

|(i|-2“2^ 

|a| < 1, jdj > 1, 

|n- 2 n 2 ^ |c| > 1 

a-2“i, 

|a| > 1, ladl < 1, 

> 


jaj-2“i-“2|d|-2«2^ 

a 

V 

W 

a. 

V 


where ui := Re( ^^ 2^ '*^2 •= ^ ). Note that most these contributions are 

already visible in fIM)) . Moreover, as in fl^ we have |Ps(t)| = |ap“i|6|“i|c|“2|(i|“i+“2 
Next, we consider the quantity which appears in the integral fl^ . 

Using |CS 80 j or [LM 09 ] in the nonarchimedean case, or [W88] . and |W 92 j as expli¬ 
cated in [S 93 ] and |S 95 ] in the archimedean case, we have 

X! ^x{(^d,bc,bd,cd)x{t), ( 42 ) 

X€Xtt 


where is a finite set of hnite functions depending on the representation tt, and 
$2, is a Bruhat-Schwartz function C for each x. 

Thus we obtain a sum of integrals of the form 

f $(ad, be, bd, cd)x(f)|a|^l“l+^l“dDfc 2 niH 2 n 2 | Jfc3niH3«2U|fc4niH4«2 ^ 3 ) 

JD 

where D is one of our 16 subsets, fci,..., /c4 and h,... , 1 ^ are explicit integers depend¬ 
ing only on D, $ is a Bruhat-Schwartz function ^ M, and x is a real-valued hnite 
function Z\T M. 

Now, for each of the seven cases which appear in fl 54 D . make a change of variables, 
as in the unramihed case so that |a|*^i“i+*i“ 2 | 5 |^ 2 “i+^ 2 “ 2 |c|^ 3 WiH 3 U 2 |(^|fc 4 «iH 4 U 2 jg expressed 
as powers of the absolute values of the new variables. For example, when \d\ > 1 and 
|a|,| 6 |,|c| < 1, we have |z2s(t)||d|“^®Ui)-Re(s2)+4 _ |a| 2 “i| 5 |“i|c|“ 2 |c?|“i-“ 2 ^ substitute 
a' = ad,b' = bd,c' = cd and d' = d~^. After the substitution, each exponent is a 
nontrivial non-negative linear combination of ui and U2. Also \d'\ is bounded, and 
we have ^{a',b'c'{d'Y,b',c'), which provides convergence as |a'|,|fe'| or \c'\ -> oo. It 
follows that the integral converges provided Ui and U2 are sufficiently large, relative 
to the hnite function x. 
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As a second example, we consider the case |a|,|c| > l,| 6 |,|(i| < 1 . In this case, we 
make the change of variables b' = be, d' = acd, a' = a~^,c' = c~^. We obtain the integral 

J da'dU dc'dd', 

assuming that Ui and U2 are positive and sufficiently large (depending on x ), the in¬ 
tegrals on a' and c' are convergent due to the domain of integration, and the integrals 
on b' and d' are convergent from the decay of $. Indeed, ^{c'd',b',a'd',a'{b')‘^c'd') « 
\a'b'c'{d'Y\~^ for any positive integer N because $ is Bruhat-Schwartz, and then 
\a'b'c'{d'Y\~^ < \b'd'\~^ on the domain D. The other hve cases appearing in 0341 ) are 
handled similarly. 

The nine cases which do not appear in flHTl) are easier. For example suppose that |c| 
and |d| are both > 1 while |a| and \b\ are both < 1 . Then the exponents of |a| and | 6 | are 
the same as in Ps(t), i.e., they are 2 ui and Ui respectively. This gives convergence of 
the integrals on a and b when Re(tii) is sufficiently large (relative to x). The integrals 
on |c| and |d| converge because of the rapid decay of <F in cd. The other eight cases 
are treated similarly, completing the proof of the convergence of /i(VF,/, ^; s). Now 
consider Ii{R(k).W, R{k).f,uj^{k). 4 >-, s). Each bound used in the analysis of R can be 
made uniform as k varies in the compact set k. Hence Ii{R{k).W, R{k).f, u^{k).(j); s) 
varies continuously with k so its integral is again absolutely convergent. □ 

6 . 2 . Continuation to a slightly larger domain. In this section, we prove that 
the local zeta integral I{W, /, s) extends analytically to a domain that includes 
the point si = 5 ,S 2 = 1 . This point is of particular interest for global reasons. We 
keep the notation from the previous section. There are two issues. The hrst is 
related to the convergence of the integral E2-fx,s- we have seen, this integral is 
not absolutely convergent at ( 5 , 1 ). We must show that it extends holomorphically to 
a domain containing ( 5 , 1 ). Then we need to prove convergence of the integral over 
Z\T. The domain of absolute convergence for this integral depends on the exponents 
of the representation tt. To make this precise, we use terminology and notation from 
|AS 06 j . section 3 . 1 . 

Proposition 6.6. Suppose that H satisfies H{dR) and that ti and T 2 satisfy 11 ( 62 ) 
(as in |AS 06 j . section 3 . 1 ). Then for any e > 0 , the local zeta integral I(W,f,(j);s) 
extends holomorphically to all s € satisfying Re(si - S2) > max( 26*4 + 2^2 + 2 , 3 ) + 
E, Re(si + S 2 ) k 5 + £, Re(s2) k ^ + £, Re(si + 2 ^ 2 ) k 202 T 

Proof. We hrst need to extend E2-fx-,s beyond its domain of absolute convergence. 
It suffices to do this for hat iT-hnite sections, even though the convolution sections 
encountered in section 15.21 are not, in general, hat of iF-hnite. Indeed, the integral 
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operator E 2 commutes with the convolution operators considered in 15.21 Moreover, 
these operators are rapidly convergent, and hence preserve holomorphy. 

As we have seen in the unramihed computation E 2 can be expressed 
where E3 is an operator dehned on Ind^^((x; by the U2 integral in ([ 32 ]) • 

Then, M(w[^,x',s) is absolutely convergent for Re(s2) > |,Re(si - S2) > l,Re(si + 
S2 - 3) > 3. If we insert absolute values into the integral which dehnes E 3 , we obtain 
a standard intertwining operator attached to We may write is as a composite of 
rank one intertwining operators attached to {a > 0 : < 0}. The rank one operator 

attached to a is absolutely convergent provided that is positive. Running 

through the eight relevant roots, we hnd that only one rank one operator diverges 
at (5,1). It is attached to the simple root as which satishes = 2Re(s2) - 2. 

Thus, we only need to improve our treatment of the integral over a single one- 
parameter unipotent subgroup. Thus, we consider 

f^&u’[3]x34(r)g)^(c4r)dr, (44) 

where C4 € F and is a section of the family Ind p„((Y; s e C^. Notice 

that f|441) may be regarded as a Jacquet integral for the rank-one Levi attached to 
the simple root 03. By |J67] . this extends to an entire function of s when is 
flat. If we apply it to the output of M{w^^,x]s), then it has no poles other than 
those of M{wi^,x] s). Now we use again the fact that the asymptotics of a Whittaker 
function, are controlled by the exponents of the relevant representation. This time 
we apply it to the induced representation of our rank one Levi. For most values of s, 
the exponents are ((x; and ((x; and the Whittaker function 

is bounded in absolute value by a linear combination of spherical vectors. 

On the line S 2 = 1, this may fail: if {{x'l = iiX'i then an 

extra log factor appears in the asymptotics of the Whittaker function (cf. |GHllj . 
6.8.11, for example). Bounding log|x| by |x|“^ with £ > 0 as x ^ 0, in this case, 
we again bound the integral 0441) by a sum of spherical sections. In fact, the extra 
|a;|^ may be safely ignored, since we obtain convergence for s in an open set and 
£ > 0 can be taken arbitrarily small. Thus, if W 2 = w[ 3 ]w 2 , then E 2 .fx;s extends 
holomorphically to the domain where the standard intertwining operator attached 
to w '2 converges on both and /Re(.),^i^[3]- Inspecting {a > 0 : {w'^y^a < 0}, 

we see that this means Re(2s2 - l),Re(si - S2 - 3) and Re(si + S2 - 5) must all be 
positive. As a side effect, we hnd that \E 2 .fx-,si 9 )\ is bounded by a suitable linear 
combination of M(wywy,Re(s )).and Re(s))./^g^^j. 

As before, we obtain a sum of integrals of the form 0431) where now the integers 
ki,... ,k 4 and /i,..., ^4 depend on the choice of domain D and on a choice of between 
W 2 and W 2 . 
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In order to obtain a precise domain of convergence, we need information about 
the finite function x. Firstly, since we have taken absolute values and assumed uni¬ 
tary central character, it factors through the map t ^ (|a(i|, |6c|, |6(i|, |cci|). We may 
assume that x is given in terms of real powers of the coordinates and non-negative 
integral powers of their logarithms, since such functions span the space of real-valued 
hnite functions. Since a power of log y may be bounded by an arbitrarily small pos¬ 
itive (resp. negative) power oi y a.s y ^ oo (resp. 0), for purposes of determining 
the domain of convergence, we may assume that there are no logarithms. Thus 
we may assume x{t) = |ad|^i|6c|/’2|fe(i|P3|c(i|P4 with pi, P 2 , Ps, Pa ^ 1^- The quadruples 
{pi, P 2 , Ps, Pa) which appear are governed by the exponents of tt, by |CS80] or [LM09] 
in the nonarchimedean case, and |W88j. [W92] fsee also IHSl) in the archimedean 
case. Hence they are bounded in absolute value by max(6*2,04), by the definition 
of H{92) and H{94) in |AS06j and the bound on exponents of tempered represen¬ 
tations found in |W92j . Theorem 15.2.2 in the archimedean case, or |W03j in the 
nonarchimedean case, we see that |pi|, |p2| ^ ^4, IdsI, |p4| < 

What remains is a careful case-by-case analysis along the same lines as the proof 
of convergence. For each choice of H, after a suitable change of variables we have an 
integral which is convergent provided ui and U 2 are sufficiently large, and “sufficiently 
large” is given explicitly in terms of pi,..., p4. 

For example, the above integral corresponding to the case |a|, |c| > 1 and |6|, |(i| < 1 
will now feature a Schwartz function integrated against 

and so will converge provided ui + U2 + Pa + P4, + p 2 + ps, 2mi + pi + 2 p 2 , and 

ui + U2 + Pi + Ps + Pa are all positive □ 


7. Local zeta integrals II 

In this section we continue our study of the local zeta integral /(hF, /, s) at the 
ramified places. Write Up for the unipotent radical of the parabolic opposite P. 
Notice that PUpW is a Zariski open subset of GSOi 2 . We say that / € Flat(y;) is 
simple if it is supported on PKi where Ki is a compact subset of UpW. 

Proposition 7 . 1 . Suppose that f is simple. Then I{W, f, cj)] s) has meromorphic 
continuation to for each cf e i5(Mat4x2) and each W e ^^^(Tr) Moreover, if Sq is 
an element o/C^, then there exist W, f and (f) such that I{W,f,(j)] sq) 0. 

Proof. We begin with some formal manipulations which are valid over any local field. 
The process requires many of the same subgroups which were defined during the proof 
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of theorem 13.11 and we freely use notation from that section. 

= r f f W{g)f{wug,s)[u^{ug).^ 
JzUi\CQ Ju^\Uq -JMatiK2 





(oi 


0 


) dr du dg. 


Define Ui{a,b,c) as in 


Then 

f j, 

[uj^{ug).(j)] 


h \ = [oj^iUi{ri,r2,c)ug).(j)](Eo), 
0 


(for any c) where Sq := ^^ 2^ and r = (ri r2). Also hT(17i(ri,r2,c)(7) = 'ilj{ri)W{g) 
1 


ijj(r 


)W{g). Hence 


f f W{g)f{wug,s)[u^{ug).^](Eo)dudg, 
Jzu5\Cq Ju^\Uq 


where is the product of Uq, U 2 and the center Z{Ui) of Ui. 

Recall that Cq is a standard parabolic subgroup of Cq. Let U(Cq)~ denote the 
unipotent radical of the opposite parabolic. Then Cq ■ U{Cq)~ is a subset of full 
measure in Cq and we can factor the Haar measure on Cq as the product of (suitably 
normalized) left Haar measure on Cq and Haar measure on U{Cq)~. Hence 

I{W,f,(j)-,s)= / / / W{gui)f{wugui,s)[u^{ugui).(j)]{Eo)duduideg 

JZUf\C^ Ju(c^)- Ju^\Uq 


Conjugating g across u, making a change of variables, and making use of the equiv- 
ariance of / yields 

I(W, f, (f)] s) = / / J(R(ui).W,uj^(uui).(j)-,s)f(wuui,s)dudui. 

Ju{c^)- Ju^\Uq 


where 


J{W,(j),s)= / lH(5f)[a;^(5f).0](So) :ia.ci{g){xc){wgw dig. 

JzusXc^ 


with Jaci((7) being the Jacobian of the change of variables in u. Now conjugation 
by w maps Uq\Uq x U{Cq) isomorphically onto the unipotent radical Up of the 
parabolic opposite P. Hence, if $ is any smooth compactly supported function on 
Uq\Uq X U{Cq)~, then there is a flat section / such that f{wuui,So) = <h(u,ui). 

We claim that the integral J(W, 0; s) converges provided Re(si-S2) and Re(s2) are 
both sufficiently large, and that J{R(ui).W,u.^{uui).(f)-, s) extends meromorphically 
to and is a continuous function of uui away from the poles. Granted this claim. 
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is clear that if the integral of Sq) against the arbitrary test 

function f(wuui,So) is always zero, then J(W, (f)] Sq) is zero for all W and 0 . 

Now, Cq = {Pi X ^2)° is the intersection of Cq with the product of the Klingen 
parabolic Pi of GSp^ and the Siegel parabolic P2 of GSO4. Let C = {Pi x M2)° 
denote the subgroup of elements whose GSO4 component lies in the Levi, and let 
= G' n U5 = UqZ{Ui). Then G' surjects onto ZU^\Gq, which is thus canonically 
identihed with ZU^\G'. Expressing the measure on Gq in terms of Haar measures 
on Ui,U2 and (Mi x M2)°, and then identifying Z{Ui)\Ui with Mat2xi(-E) via the 
map 'Ui([ri r2]) = t/i(ri,r2, 0 ), yields the following expression for J{W,(l)-,s): 


( 45 ) 

Now, elements of G' map under j into the Siegel Levi of Spi^. So that 


lVIatix2 (MixM2)° 


where 6c^, is the modular quasicharacter. 


[u^oj{c').(j)]{^) = |det c'|20(^-c'), 


where • is the rational right action of G' on Mat4x2 by 


^■mQ{9i,92) =91^^92- 

(with TUg as in (jl])). The stabilizer of the matrix Hq is precisely the group M5 
introduced in the proof of theorem 13.11 

In fHS]) . conjugate m across Ui{r), make a change of variables in r, and let Jac2(m) 
denote the Jacobian. Dehne 


9s{'fn) = (xi s){wmw ^)J(^i(m)| det mja Jaci(m) Jac2(m). 

Then replace m by rn^m'^{g) where € M5 and m'^{g) = m{l, I2, g), (with 
(EU)). Observe that 


Eo ■ m5m'^{g)ui{r) 


r-g 


9 

0 


m as in 


Hence if x{g,r) = 'rn'^{g)ui{rg ^), then 


r\ 

J(VL,0;s) = If J'{R{x{g,r)).W,s)ct>\g\ 

Matix2GL2 \0/ 


9s{m'5{9))\det g\ ^dgdr, 


where 


J'{W,s):= J W{m5)ps{m5) 

ZUe\M5 


dm^. 


Direct computation shows that 9s{'m'^{g)) = |det5f|®2y2(det^f). 


( 46 ) 

( 47 ) 
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Write M5 = UqTqK^, where T5 = T n M5 and is the maximal compact subgroup 
of the GL 2 factor. Then 

J J Z\T^ 


where 5b^ is the modular quasicharacter of the standard Borel subgroup of M5. 
Set = diag(a, 1,1, a“^, 1,1,1,1,a, 1,1,a~^), and write t a,s for € Tg 

and a e F^. Then 


J\W,s)= f f J"{R{t'^{a)k).W,s)M{o))dt, 
Jks Jf^ 

where J"(IT, s) = f dtf^. 

Jz\T 6 


IZ\Te 

Observe that J'(W,s) may be written formally as 


f r{Rig,).W,s)fisigi)dgi. 

J Me\Ms 


Also, direct computation shows that ps(f^a)) = |a|^i ^yi(a)/x2(o)- 

For 01 a smooth function of compact support ->■ C let 

[01 x-i VF](5() = f W{gUi{ri,r2,0))(f){ri,r2)dr. 

J 

Observe that 


[01 =.1 lT](M5(t,^3)) = fF(M5(f, (?3))0l(i73' • 


1 

0 


fdet ^3). 


Thus, by replacing VF by 0i x-i VF (which is justified by |DM78] ) we may introduce 
what amounts to an arbitrary test function on Mq\M^. Hence 


J'(W,So) = 0VhF » J"(hF,So) = OVso. 


Similarly, if 

[(t^2*2W]{g)-.= f W{gue)hMdue, 

JUe 

then J"(02 *2 hF, So) = OV02 e C^{Uq) if and only if IT vanishes identically on Tg. 
In particular, if J"(1 T,Sq) vanishes identically on Wp^(7r), then is trivial- 

a contradiction. 

This completes the formal arguments for proposition 17.11 It remains to check the 
convergence and continuity statements made above. These will be proved based on 
facts about asymptotics of Whittaker functions and the Mellin transform 

Zg „.$(«)= f <l>(x)^(a:)(log|x|)’"bFd’'x, (48) 

where <F e i5(F), ^ C, a character, n > 0 € Z, « e C. We recall some properties. 
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Proposition 7.2. Fix a character ^ and a non-negative integer n. 

(1) There is a real number c depending on ^ such that the integral defining 
converges absolutely and uniformly on {m e C : Re(n) > c + e} for all e > 0 and 
all $ e 5(F). 

(2) There is a discrete subset of C such that Z^ n-^ extends meromorphically 
to all of C with poles only at points in S^. Moreover, there is an integer o^,„ 
such that no pole of Z^ n-^ has order exceeding o^^n, for any 

(3) If F is archimedean, then Z^ n-^ = for some rational function Q. 

(4) We have 

(49) 

Proof. If n = 0 then the hrst three parts are proved in |T67j . Convergence for n > 0 
is straightforward, since log |x| grows slower than any positive power of |a:| at inhnity, 
and slower than any negative power as |x| 0. Eqnation (H9ll is clear in the domain of 

convergence, and follows elsewhere by continnation. The hrst three parts for general 
n then follow. □ 

Next, we need a version of the expansion flT2]) . Specihcally, if we replace W{t) by 
W{tk) then each $ 3 ; in (l42l) will be in S{F‘^ x K). (See jS93j . especially the remark 
on p. 20 .) 

Let ns now consider the convergence issnes raised by onr formal compntations 
more carefnlly. Recall that I{W, f, s) was initially expressed as an integral over 
Uq\Uq X ZUf\CQ. In the conrse of onr argnments, we have expressed it as an iterated 
integral over 


{U^\Uq X U{Cf^)-) X (F2 X GL2) X (FX X K,) X Z\T,. 

In order to perform the integration on Z\Tq we may identify with with {tQ(a) = 
diag(l,l,a■^a-^l,a-^l,a■^l,l,a-^a-l),a e F’^}. Then = |a|^T^“^^^(a). 

Now, the integral J"(W, s) is the Mellin transform taken along the one dimensional 
torns we nse to parametrize Z\Tq. Its convergence and analytic continnation follow 
directly from proposition 17.21 and fl42]) . Now consider J"{R{t'^{a)k).W,s) with a e 
F^,k e F 5 . We claim that it is smooth and of rapid decay in a. In the domain of 
absolnte convergence, this is easily seen. For s ontside of the domain of convergence, 
we use (14^ to pass to the Mellin transform of a suitable derivative at a point inside 
the domain of convergence. To get J'{W, s) we integrate k over the compact set k and 
take another Mellin transform in the variable a. This of course converges absolutely, 
and by similar reasoning, we see that J'{R{x{g,r).W),s) is smooth. Now, set g 
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equal to ( o *2 ) ^ consider the integral 

r\ 

9 1 dk dbdrd^ti d''t2, 

0/ 

where S 4 and S 5 are two more complex variables, and Kgl 2 is the standard maxi¬ 
mal compact subgroup of GL 2 . The integrals on k,r and b converge absolutely and 
uniformly because Kgl 2 is compact and 0 is Schwartz-Bruhat. The integrals on ti 
and t 2 take two more Mellin transforms, yielding a meromorphic function of four 
complex variables. The restriction to a suitable two-dimensional subspace of is 
J{W,(j),s). Moreover, J(R{ui).W,u^{uui).(j), s) remains continuous in ui e U{Cq)~ 
and u e Uq\Uq, which completes the proof. □ 


I 111 I J'W<9,r)).W,s)<P 


8. Global Identity 


We now return to the global situation. Thus F is again a number held with adele 
ring A, while and i/jn, and Flat(x), are dehned as in section [3] and [2|, respectively. In 
addition, let tt = be an irreducible, globally i/^Ar-generic cuspidal automorphic 
representation of GSp 4 {A) x GS 04 {A), with normalized central character oJtt, and (p 
be a cusp form from the space of tt, etc. 

For r a representation of ^G dehne L{u,TT,r x rf) to be the twisted L function. 
Thus at an unramihed place v the local factor is 

L^{u,TT^,rxr]^) = det(/-g"“?7^(tn„)r(r^J)“\ 

where to^ is a uniformizer, is the cardinality of the residue class held, is the 
semisimple conjugacy class attached to vr^, and rjy is the local component of rj at v. 


Theorem 8.1. Suppose that ^ Flaf(x))0 = Ilv4’v ^ i5(Mat4x2(A)) and 

IT^ = Wv^v (the Whittaker function of n as in theorem \3.1\ are factorizable. Let 
difxv-,s,^v,(f>v) be the local zeta integral, defined as in (125]) . and let S be a finite set 
of places V and all data is unramified for all v i S. Then for Re(si - S 2 ) o,nd Re(s 2 ) 
both sufficiently large, the global integral yj,0), defined as in ([7|), is equal to 




1, TT, St^gp^ (2) St^pi) 


X3 

xixi 




2.»r,S*GSw®‘5i^LS“> 


X3 ~ 
X1X2 ' 


NS{s,x) 


times 


veS 


where N^{s,x) is the product of partial zeta functions corresponding to (1331) 









TWISTED SPINOR L-FUNCTION 


41 


Remark 8.2. Let rji and r ]2 be any two characters of Fix vr and let he its 

central character. Then the system 


A1A2 

X 3 


— C^TT^ 


X3 

Xixl 


Vi, 


X3 

X 1 X 2 


V 2 


has a unique solution. If rji = r ]2 is trivial, then it is given by Xi = X 3 = X 2 = 1- 


Proof. It follows from theorem 16.II the bound obtained in [L71] that for any cuspidal 
representation vr = of GSp 4 ^{A)xGS 04 .{A) the inhnite product fKj A) 

is convergent for Re(si - S 2 ) and Re(s 2 ) sufficiently large. It then follows from theo¬ 
rem [XIl and the basic results on integration over restricted direct products presented 
in |T67j that 


iUx;s.tA) = Y\iUx.;s,w,M. 


which, in conjunction with theorem 16.II again gives the result. 


□ 


Corollary 8.3. Let vr^ he the local constituent at v of a cuspidal representation tt. 
Then the local zeta integral s) has meromorphic continuation to €? for 

any Wy, fy and (j)y. 


Proof. This follows from a globalization argument. We create a section of the global 
induced representation which is / at one place and simple at every other place. Mero¬ 
morphic continuation of the global zeta integral follows from that of the Eisenstein 
series. Having shown meromorphic continuation at every other place in proposition 
17.11 we deduce it at the last place. □ 


9. Application 

In this section we give an application relating periods, poles of L functions, and 
functorial lifting. The connection between L functions and functorial lifting in this 
case was obtained in |AS06] . 

Let n be a globally generic cuspidal automorphic representation of GSp^, and let 
Ti and T 2 be two cuspidal automorphic representations of GL 2 . Assume that H, ti and 
T 2 have the same central character. Then ti ® T 2 may be regarded as a representation 
of GS '04 via the realization of GS '04 as a quotient of GL 2 x GL 2 discussed in section 
El and when H ® ri ® r 2 is restricted to the group Gq (which we identify Gq with 
subgroup of GSpi x GSO/^ as in section E] its central character is trivial. 

Now take Si and S 2 to be two complex numbers. Let Xi = X 2 = Xs be trivial. 
Consider the space Flat(x) of flat sections as in sectional Its elements are functions 
X G'(A) ^ C, but we regard each as a function x G'(A) ^ C by pulling it back 
through the function (si,S 2 ) (si, S 2 , Then theorem 18.11 relates the global 
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integral ([7]) with the product of L functions 


SfSi-S2-2 ~ N r5/®i '®2 ~ 4 ~ 

L { —z—,nX—-—,nXTs). 


For / € Flat(x), let 




lim 

^-(5,1) 


/ gl + 52 -4 

V 2 



■Sl ~ -52 ~ 
2 


2 



be the residue of the Eisenstein series at (5,1). As / varies we obtain a residual 
automorphic representation which we denote IZ. Given i^lZ and (j) e i5(Mat4x2(A)), 
we define the Fourier coefficient r^G) exactly as in [Hi Varying r and (j) we obtain a 
space of smooth, iF-finite functions of moderate growth Z(A)Cq(F)\Cq{A) ->■ C. 
We denote this space FC{TZ). Write Vn for the space of the representation fl and W 
for that of r. Then, define the period "P : Vi x W ^ FC{1Z) ^ C, by the formula 

= f f^'^H9)^u{gi)^r{92)dg. 

Jz\Cq 

Theorem 9.1. First suppose that ti 4^ T 2 . Then the following are equivalent: 

(1) L^{s, n X n) and L^{s, fl x T 2 ) have poles at s = 1. 

( 2 ) n is the weak lift of Ti x t 2 

(3) the period V does not vanish identically on Vn W FC{TZ). 

Similarly, if ti = T 2 , then the following are equivalent: 

( 1 ) L^{s, n X Ti) has a pole at s = 1 . 

( 2 ) n is the weak lift of Ti x t' for some cuspidal representation t' of GL 2 {A), 

(3) the period V does not vanish identically on Fq ^ W ^ FC{fR). 


Proof. The relationship between poles and the similitude theta correspondence was 
established in |AS06j . What is new here is the period condition, which follows from 
our earlier results. Indeed, for / e Flat(x),^ e i5(Mat4x2(A)) ^ Fi and ipr e W, 

the period P((pn, V?t, vanishes if and only if 


lim 

s^(5,l) 


/ 5i + 52 -4 

I 2 



■Sl ~ -52 ~ 
2 


2 



(pn,V?r,0) * 0. 


By |AS06] . the local components of 11 all satisfy 17(15/34) and the local components 
of r all satisfy 77(1/9). Hence each ramihed local zeta integral is holomorphic at (5,1) 
by proposition 16.61 Moreover, by proposition 17.11 each ramihed local zeta integral is 
nonzero at (5,1) for a suitable choice of data. The result follows. □ 
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10. A SIMILAR INTEGRAL ON GSOis 

In this section we consider the global integral ([ 7 j) in the case n = 3. Onr unfolding 
does not produce an integral involving the Whittaker functions attached to our cusp 
forms, but it does reveal another intriguing connection with the theta correspondence. 

As before, the space of double cosets P\GSOis/Rq is represented by elements of 
the Weyl group, and 

= E where 

w^P\GSOis/Rq 


iv,{fx,s.^A) = f ^{g) f fxA^'^‘^ 9 ) I uiU2g) dui du2 dg, 

C^(F)\Cq(A) U^(A)\Uq(A) [U^] 

which is zero if is nontrivial, or if some parabolic subgroup of Gq stabilizes 

the flag 0 c Tf^ c i'^ RqI where Tf^ is the image of Uq and (JTqY is 

its perp space relative to the symplectic form dehned by composing I: Z(Uq) ->■ Ga 
with the commutator map Uq!Z{Uq) Z{Uq). 


Lemma 10 . 1 . Let wg denote the longest element of the Weyl group of GSOig, let 
Wi be the shortest element of {W n P) ■ w^- (W n Q) and let W 2 = Wi ■ zc[32]. Then 
Pw 2 Rq is a Zariski open subset ofGSOis- 


Proposition 10 . 2 . I^(fy..s,Lp,(l)) is zero unless w is in the open double coset. 


Proposition 10 . 3 . Let Uq c Gq be given by 


' 

/ 

/I 

Xi 

X 2 

Xq 

X 4 

X5\ 


n 


Xj 

Xs 






1 

0 

0 

Xq 




1 x'^ 

Xg 

* 



Uq{x,x') := 




1 

0 

1 

0 

0 


5 


1 

0 

1 

-Xg 

-Xq 









1 






1 

-Xi 



\ 

\ 





1} 


\ 




1 



x'eGl 


where entries marked x- are determined by symmetry, and for x e and x' € A^, let 


i>Uo{uo{x,x')) =Y{.Xi +Xq-x[ -Xq + Xq). 

Let SLY be the copy of SL 2 generated by U^a^i and let Rq be the product of Uq 
and SLY- Let Yrq be the character of Rq which restricts to ipUo and to the trivial 
character of SLY- Let 

= J Lp(rc)YRo(,r)dr= J j ip{uhc)Yuo{'^) dhdu, (c€C'q(A)). 

[Ro] [Go] [slY] 
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Let V 4 = {u(x,x') : Xi = x',i = 1,4,6} c [/q, and let 


eo 


/I 0 0\ 
0 1 0 
0 0 0 
0 0 0 
0 0 1 


€ Mat6x3(-^)- 


\0 0 0 / 


Then 

= J (^(^0’^«o)(c) J f^.siw2Uc)[u^{uc).(j)]{^o)dudc. 

Z(A)V4{A)\Cq{A) U’^2(A)\U(A) 


Remark 10.4. It was shown in |GRS97j that the period we obtain in the GSp^ 
characterizes the image of the theta lift from SOq to SpQ. 


Proof. First, Uq^ is the set of all ^^(0,^,0) such that rows 2, 5 and 6 of R are zero. 
It follows that 

e^Q\^,uiU2g)-= J 9{(j),uiU2g)dui = Y,[‘^A'^29)-(p]{0, 


where the sum is over f € Mat 6 x 3 (F’) such that rows 3,4 and 1 are zero. Next, the 
identihcation of Cq with a subgroup of GSp^ x GSOq identihes Gq'^ with the subset 
of elements of the form 



(t Xi 

X 2 

X 3 

X 4 

xd 



a 

0 

0 

h 





a' 

b' 

0 


(g W ] 



d 

d' 

0 


’ 1 


c 



d 




\ 




t\) 



, t € GL \, 



e GL2, g £ GL3 


(50) 

Now we can expand (p along the abelian unipotent subgroup which consists of 
elements of the form ui{W) := (/e, (^ )). W' e ^A 3 . The constant term is of 
course zero. The group Gq^ acts transitively on the nontrivial characters. As a 
representative for the open orbit we select the character 'f’ 2 ,i{.Ui{W)) := ' 0 (VF 2 ,i)- 
The stabilizer of this representative can be described in terms of the coordinates 
from flSU]) as the set of elements of Gq^ such that g € GL^ is of the form () with 
gi € GL 2 and det^^i = A. Denote this group by Gf^. Now we write the integral as a 
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double integral, with the inner integral being 


/ / 

■A3]n“2(A)\C7 


r r’^2 

(it),U 2 Ui{W)g) du^i\\(w) d[¥. 


(51) 


Now, 

nQ(e,0,0)ni(lh^) = ni(Vh)nQ(e,0,0)nQ(0,ei^,-ei^iO, (W € ^Ag, ^ ^ Matg.s). 
It follows that (1^ is equal to 

J J fxAw2U2g)YX^i>{'^2g)4]{0M^'W ti)'il:2\{W)du2dW, 


with ^ summed over 6x3 matrices such that rows 3,4 and 6 are zero. Clearly the 
integral on W picks off the terms such that t 0'*/’2 i(^) trivial. Now, direct 

calculation shows that 




/6 6 esX 

^4 ^6 

0 0 0 
0 0 0 ’ 

^7 ^8 ^9 

\0 0 0 / 


w 


'yi y2 0 \ 

?/3 0 -y2 

^0 - 2/3 -yi) 


i 




det 


\ 


hz 

U 

\ii 


-y\ 

^5 

^8 



So, in the coordinates above, the condition for i(^) tie trivial is 


^4 = ^7 = 0 and det 


^8 ^9; 


= 1. Observe that if is also zero, then the function 


g ^ [oo^pig) is invariant on the left by {{!&, (“ ^u -^)) e Cq'^ ■ u = (^ ^ ^ C-^a}- 

Thus, the contribution from such ^ is trivial by cuspidality. The group permutes 
the remaining terms transitively, and the stabilizer of is 


cr ■■= 


i (t xi X2 xs X4 a;5\ 

a 0 0 6 X- 

a' b' 0 X- 


d' 


0 

d 


>h 

tA/ 


(g W ' 
I 


(t 


: 


g = 



Expanding first on Xi and X 4 , and then on the unipotent radical of the diagonally 
embedded GL 2 , and using lemma 1X71 two more times gives the result. □ 
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